CODEWITHSUNDEEP
stringtokenize
Pladnius Brooks
Pladnius Brooks

Reputation: 1308

Good way to tokenize line from file without using external libraries?

I am trying to tokenize a database dump separated by commas. I only need to read the first word, which will tell me if this is the line I need and then tokenize the line and save each separated string in a vector.

I have had trouble keeping all of the datatypes in order. I use a method of getline:

string line;
    vector<string> tokens;

// Iterate through each line of the file
while( getline( file, line ) )
{
    // Here is where i want to tokenize. strtok however uses a character array and not a string.
}

The thing is, I only want to continue reading and tokenize a line if the first word is what I am after. Here is a sample of a line from the file:

example,1,200,200,220,10,550,550,550,0,100,0,-84,255

So, if I am after the string example, it goes ahead and tokenizes the rest of the line for my use and then stops reading from the file.

Should I be using strtok, stringstream or something else?

Thank you!

Upvotes: 0

Views: 393

Answers (3)

CATspellsDOG
CATspellsDOG

Reputation: 109

Tokenizer.h

#ifndef TOKENIZER_H
#define TOKENIZER_H

#include <string>
#include <vector>
#include <sstream>

class Tokenizer
{
public:
    Tokenizer();
    ~Tokenizer();
    void Tokenize(std::string& str, std::vector<std::string>& tokens);
};

#endif /* TOKENIZER_H */

Tokenizer.cpp

#include "Tokenizer.h"

using namespace std;

string seps(string& s) {
    if (!s.size()) return "";
    stringstream ss;
    ss << s[0];
    for (int i = 1; i < s.size(); i++)
        ss << '|' << s[i];
    return ss.str();
}

void tok(string& str, vector<string>& tokens, const string& delimiters = ",")
{
    seps(str);

    string::size_type lastPos = str.find_first_not_of(delimiters, 0);
    string::size_type pos = str.find_first_of(delimiters, lastPos);

    while (string::npos != pos || string::npos != lastPos)
    {
        tokens.push_back(str.substr(lastPos, pos - lastPos));
        lastPos = str.find_first_not_of(delimiters, pos);
        pos = str.find_first_of(delimiters, lastPos);
    }
}

Tokenizer::Tokenizer()
{
}

void Tokenizer::Tokenize(string& str, vector<string>& tokens)
{
    tok(seps(str), tokens);
}

Tokenizer::~Tokenizer()
{
}

To tokenize a string

#include "Tokenizer.h"
#include <string>
#include <vector>
#include <iostream>
#include <cstdlib>

using namespace std;

int main()
{
    // Required variables for later below
    vector<string> t;
    string s = "This is one string,This is another,And this is another one aswell.";
    // What you need to include:
    Tokenizer tokenizer;
    tokenizer.Tokenize(s, t); // s = a string to tokenize, t = vector to store tokens
    // Below is just to show the tokens in the vector<string> (c++11+)
    for (auto c : t)
        cout << c << endl;
    system("pause");
    return 0;
}

Upvotes: 0

ofavre
ofavre

Reputation: 4786

#include <iostream>
#include <fstream>
#include <vector>
using namespace std;

void do(ifstream& file) {
    string line;
    string prefix = "example,";

    // Get all lines from the file
    while (getline(file,line).good()) {
        // Compare the beginning for your prefix
        if (line.compare(0, prefix.size(), prefix) == 0) {
            // Homemade tokenization
            vector<string> tokens;
            int oldpos = 0;
            int pos;
            while ((pos = line.find(',', oldpos)) != string::npos) {
                tokens.push_back(line.substr(oldpos, pos-oldpos));
                oldpos = pos + 1;
            }
            tokens.push_back(line.substr(oldpos)); // don't forget the last bit
            // And here you are!
        }
    }
}

Upvotes: 1

alien01
alien01

Reputation: 1332

How do I tokenize a string in C++?

http://www.daniweb.com/software-development/cpp/threads/27905

Hope this helps, though I am not proficient C/C++ programmer. For the record it would be nice if you could specify in the tags or in post language you are using.

Upvotes: 0

Related Questions