Tokens in C++ and how they are used

Question

In Bjarne Stroustrup's book "Programming Principles and Practices using C++", he is explaining how you can use tokens to stop C++ from automatically using the order of operations on a simple calculator. The code he has given in book does not work - either because of my stupidity, or because I am missing something. I understand that tokens is essentially breaking down lines of code into characters. For example, If I have 5 * 3 there are 3 tokens. Two values and one character. I do not know how to incorporate tokens, or why we should use them. Bjarne's example of a user defined token is this:

class Token {
    public:
        char kind; // what kind of token
        double value; // for numbers - a value
        Token(char ch) // make a token from a char using a constructor
            :kind(ch), value(0) {} // set kind to ch and value to val
        Token(char ch, double val) // make a Token from a char and a double
            :kind(ch), value(val) {}
};

In addition he has provided this example of how you can read input into a vector of tokens:

Token get_token() {} // read a token from cin

vector<Token> tok; // put the tokens in this vector

while(cin >> t5) {
        Token t5 = get_token();
        tok.push_back(t5);
    }
    return 0;

First of all... 1) He prototyped the "get_token()" function but he didn't even write any code for it. 2) Token's object 't5' is getting initialized after it's already getting read in a while loop. 3) I have tried many ways of getting cin to read input into t5, and it is not working. I even defined t5 before the while loop, and I'm getting an invalid operand error for '>>'. This is the first time in this book that I've been utterly stuck. I can't find any examples online of how you can use tokens in C++ to write programs, I only get vague definitions of what a token is. If anyone could assist me in understanding tokens or possibly point me to a good source where I can get a thorough explanation, it would be greatly appreciated.

Answer 1

Firstly, Stroustrup is not one of the great teachers, and if you are learning programming you might be better advised starting somewhere other than C++, and somewhere other than his books.

To address your question, a token is a lexical element,and exactly what it is depends on the problem you are addressing. For example, if you are dealing with arithmetic expressions:

100+2/3

then the tokens would be "100", "+", "2", "/" and "3". On the other hand, if you were dealing with English text:

"Here we go again"

then the tokens would probably be:

"Here", "we", "go", "again"

and we simply discard the space characters.

How do you "use tokens"? Well, suppose you want to write a spell-checker - you need to break up the text into words and punctuation (different types of tokens), probably discard most punctuation tokens, and then look up the word tokens in some sort of dictionary.

Answer 2

Here you are the answer to your question #1. This is the get_token() function you were asking for!

Here is a link for it below (from Stroustrup's website directly :)):

   Token get_token()    // read a token from cin
    {
        char ch;
        cin >> ch;    // note that >> skips whitespace (space, newline, tab, etc.)

        switch (ch) {
     //not yet   case ';':    // for "print"
     //not yet   case 'q':    // for "quit"
        case '(': case ')': case '+': case '-': case '*': case '/': 
            return Token(ch);        // let each character represent itself
        case '.':
        case '0': case '1': case '2': case '3': case '4':
        case '5': case '6': case '7': case '8': case '9':
            {    
                cin.putback(ch);         // put digit back into the input stream
                double val;
                cin >> val;              // read a floating-point number
                return Token('8',val);   // let '8' represent "a number"
            }
        default:
            error("Bad token");
        }
    }