CODEWITHSUNDEEP
compiler-theory
Tarun
Tarun

Reputation: 3165

how it produce multiple token for each lexeme?

I just started reading The Dragon Book and I'm finding difficulty in understanding some statements.

It says: "lexical analyser produce sequence of token for each lexeme in the source program". Can you please help me to understand the above line? I know about tokens and lexemes, but what is meant by producing multiple token for each lexeme....AFAIK LEXEME itself compromising a single token.

The complete quote is as follows:

"As the first phase of compiler, the main task of the Lexical Analyser is to read the input characters of the source program, group them into lexemes, and produce as output a sequence of tokens for each lexeme in the source program."

The above quote is from chapter 3..section 3.1 under the heading "role of lexical analyser" page number is 109

Upvotes: 0

Views: 556

Answers (1)

John Kugelman
John Kugelman

Reputation: 362087

You are correct. Tokens usually correspond one-to-one with lexemes. Try re-parsing that sentence as the "...and produce as output a sequence of tokens for the lexemes in the source program." That is the meaning the authors intended, as I read it.

Upvotes: 3

Related Questions