CODEWITHSUNDEEP
c++booststltokenize
snowdude
snowdude

Reputation: 3874

Identifying the position in the original string from a given Boost token_iterator

If a string has been processed using a Boost tokenizer is it possible to get the position in the original string that a given token iterator is pointing to:

boost:tokenizer<> tok( "this is the original string" );
for(tokenizer<>::iterator it=tok.begin(); it!=tok.end();++it)
{
    std::string strToken = *it;
    int charPos = it.?                /* IS THERE A METHOD? */
}

I realize I could create a specific char_separator with a defined list of 'kept delimiters' and specify keep_empty_tokens to try and track the progression of the iterator myself but I was hoping there was an easier way using just the iterator itself.

Upvotes: 3

Views: 1148

Answers (3)

ildjarn
ildjarn

Reputation: 62985

This appears to be what you're looking for:

#include <string>
#include <iostream>
#include <boost/tokenizer.hpp>

int main()
{
  typedef boost::tokenizer<> tok_t;

  std::string const s = "this is the original string";
  tok_t const tok(s);
  for (tok_t::const_iterator it = tok.begin(), it_end = tok.end(); it != it_end; ++it)
  {
    std::string::difference_type const offset = it.base() - s.begin() - it->size();
    std::cout << offset << "\t::\t" << *it << '\n';
  }
}

Online Demo

Upvotes: 5

Ise Wisteria
Ise Wisteria

Reputation: 11669

If you need only the end of current token, base() member function might meet the purpose:

std::string s = "this is the original string";
boost::tokenizer<> tok(s);
for(boost::tokenizer<>::iterator it=tok.begin(); it!=tok.end();++it)
{
    int charPos = it.base() - s.begin();
}

Unfortunately, there seems not to be the way to retrieve the beginning of current token in boost::tokenizer.

Upvotes: 1

Rune Aamodt
Rune Aamodt

Reputation: 2611

How about:

 int charPos = it - tok.begin() ;

Upvotes: 0

Related Questions