CODEWITHSUNDEEP
antlr
user133825
user133825

Reputation:

Difference between token definition and lexer tokens

What's the difference between defining a token in the tokens block and defining a token as a lexer index?

Upvotes: 1

Views: 1362

Answers (1)

Winston Chen
Winston Chen

Reputation: 6879

The lexer tokens are the tokens that antlr generates through parsing a certain combination of characters, for example:

fragment
F_W :   (' '|'\t'|'\r'|'\n'|'\f')*
    ;

However, the tokens in the token block are the visual tokens that users create for the benefits of parsing; they don't have a direct mappings to a set of character series. For example, 

tokens { BLOCK; }

gives you a visual token BLOCK that you can put ahead or on top of your block rule to mark the subsequest session is a block rule like this:

block: 
'{' expression* '}' ->
 BLOCK^ expression*
;

Upvotes: 3

Related Questions