CODEWITHSUNDEEP
antlrgrammar
user2202494
user2202494

Reputation: 11

No token from CommonTokenStream use Antlr

I am a newbie for Antlr. I write a little grammar for practice but get confused by the API CommonTokenStream

lexer grammar Expr;
options {
  language = Java;
}

EXPRS:EXPRT {System.out.println($EXPRT.text);};
fragment
EXPRT : 'xxx' ID {System.out.println($ID.text);} ' zzz';
fragment 
ID : ('a'..'z' |'A'..'Z' |'_' )('a'..'z' |'A'..'Z'|'_')*;

The Test code is like that:

ANTLRStringStream input = new ANTLRStringStream(msg);
Expr expr = new Expr(input);
CommonTokenStream cs = new CommonTokenStream(expr);
System.out.println(cs.size());

Whatever the input is, there is no token from the CommonTokenStream. However, it will ouput when I use expr directly. Anyone knows why? The version of antlr is 3.5

Upvotes: 1

Views: 400

Answers (1)

Sam Harwell
Sam Harwell

Reputation: 99959

CommonTokenStream is lazily initialized. To force it to immediately populate all tokens (by calling nextToken on your lexer until EOF is reached), you can call the fill() method:

CommonTokenStream cs = new CommonTokenStream(expr);
cs.fill();
System.out.println(cs.size());

Upvotes: 3

Related Questions