Python how to work with tokens

Question

I'm trying to learn about tokens in python. Basically all I know is that tokens are a way for python to recognise different objects of a python program.

I tried playing around as so:

from tokenize import generate_tokens
generate_tokens('hello this is a test string')

I got an error:

<generator object generate_tokens at 0x028D7238>

I expected a series of tuples to be displayed.

Can someone explain the concept of tokens to me and how to generate them in python? What python modules include ways to work with tokens?

Answer 1

You're making two mistakes:

• generate_tokens returns an iterable, not a list, so you need to wrap it with list() in order to display the results interactively (programmatic access wants the generator form).
• the argument is not a string, but a callable that returns a string, in the manner of file().readline

Fixed code and output in ipython so it pretty-prints lists better:

In [1]: from tokenize import generate_tokens

In [2]: from cStringIO import StringIO

In [3]: list(generate_tokens(StringIO('hello this is a test string').readline))
Out[3]: 
[(1, 'hello', (1, 0), (1, 5), 'hello this is a test string'),
 (1, 'this', (1, 6), (1, 10), 'hello this is a test string'),
 (1, 'is', (1, 11), (1, 13), 'hello this is a test string'),
 (1, 'a', (1, 14), (1, 15), 'hello this is a test string'),
 (1, 'test', (1, 16), (1, 20), 'hello this is a test string'),
 (1, 'string', (1, 21), (1, 27), 'hello this is a test string'),
 (0, '', (2, 0), (2, 0), '')]

For the next level (parsing), use either the standard ast module or the 3rd-party logilab.astng package.