CODEWITHSUNDEEP
tokenizeantlr3context-free-grammar
Jessy Diamond Exum
Jessy Diamond Exum

Reputation: 439

Antlr 3 keywords and identifiers colliding

Surprise, I am building an SQL like language parser for a project.

I had it mostly working, but when I started testing it against real requests it would be handling, I realized it was behaving differently on the inside than I thought.

The main issue in the following grammar is that I define a lexer rule PCT_WITHIN for the language keyword 'pct_within'. This works fine, but if I try to match a field like 'attributes.pct_vac', I get the field having text of 'attributes.ac' and a pretty ANTLR error of:

line 1:15 mismatched character u'v' expecting 'c'

GRAMMAR

grammar Select;

options {
  language=Python;
}

eval returns [value]
    : field EOF 
    ;

field returns [value]
    : fieldsegments {print $field.text}
    ;

fieldsegments
    : fieldsegment (DOT (fieldsegment))*
    ;

fieldsegment
    : ICHAR+ (USCORE ICHAR+)*
    ;

WS                      : ('\t' | ' ' | '\r' | '\n')+ {self.skip();};

ICHAR                   : ('a'..'z'|'A'..'Z');

PCT_CONTAINS            : 'pct_contains';

USCORE                  : '_';
DOT                     : '.';

I have been reading everything I can find on the topic. How the Lexer consumes stuff as it finds it even if it is wrong. How you can use semantic predication to remove ambiguity/how to use lookahead. But everything I read hasn't helped me fix this issue.

Honestly I don't see how it even CAN be an issue. I must be missing something super obvious because other grammars I see have Lexer rules like EXISTS but that doesn't cause the parser to take a string like 'existsOrNot' and spit out and IDENTIFIER with the text of 'rNot'.

What am I missing or doing completely wrong?

Upvotes: 0

Views: 439

Answers (1)

Mike Lischke
Mike Lischke

Reputation: 53337

Convert your fieldsegment parser rule into a lexer rule. As it stands now it will accept input like

"abc      
_     abc"

which is probably not what you want. The keyword "pct_contains" won't be matched by this rule since it is defined separately. If you want to accept the keyword in certain sequences as regular identifier you will have to include it in the accepted identifier rule.

Upvotes: 1

Related Questions