CODEWITHSUNDEEP
antlr
my_question
my_question

Reputation: 3235

How to use similar lexers

I have the following grammar:

cmds
    : cmd+
    ;

cmd
    : include_cmd  |  other_cmd
    ;

include_cmd
    : INCLUDE  DOUBLE_QUOTE  FILE_NAME  DOUBLE_QUOTE
    ;

other_cmd
    : CMD_NAME  ARG+
    ;


INCLUDE
    : '#include'
    ;

DOUBLE_QUOTE
    : '"'
    ;

CMD_NAME
    : ('a'..'z')*
    ;

ARG
    : ('a'..'z' | 'A'..'Z' | '0'..'9' | '_')+
    ;

FILE_NAME
    : ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '.')+
    ;

So the difference between CMD_NAME, ARG and FILE_NAME is not large, CMD_NAME must be lower case letters, ARG can have upper case letter and "_" and FILE_NAME yet can have ".".

But this has a problem, when I test the rule with - #include "abc", 'abc' is interpreted as CMD_NAME instead of FILE_NAME, I think it is because CMD_NAME is before FILE_NAME in the grammar file, this leads to parsing error.

Do I have to rely on such technique as predict to deal with this? Is there a pure EBNF solution other than relying on host programming language?

Thanks.

Upvotes: 0

Views: 60

Answers (1)

user1201210
user1201210

Reputation: 3809

But this has a problem, when I test the rule with - #include "abc", 'abc' is interpreted as CMD_NAME instead of FILE_NAME, I think it is because CMD_NAME is before FILE_NAME in the grammar file, this leads to parsing error.

The set of all valid CMD_NAMEs intersects with the set of all valid FILE_NAMEs. Input abc qualifies as both. The lexer matches the input with the first rule listed (as you suspected) because it's the first one matched.

Do I have to rely on such technique as [predicate] to deal with this? Is there a pure EBNF solution other than relying on host programming language?

It depends on what you're willing accept in your grammar. Consider changing your include_cmd rule to something more conventional, like this:

include_cmd : INCLUDE STRING;

STRING 
    : '"' ~('"'|'\r'|'\n')* '"' {String text = getText(); setText(text.substring(1, text.length() - 1));}
    ;

Now input #include "abc" turns into tokens [INCLUDE : #include] [STRING : abc].

I don't think the grammar should be responsible for determining whether a file name is valid or not: a valid file name doesn't imply a valid file, and the grammar has to understand OS file naming conventions (valid characters, paths, etc) that probably have no bearing on the grammar itself. I think you'll be fine if you're willing to drop rule FILE_NAME for something like the rules the above.

Also worth noting, your CMD_NAME rule matches zero-length input. Consider changing ('a'..'z')* to ('a'..'z')+ unless a CMD_NAME really can be empty.

Keep in mind, too, that you'll have the same problem with ARG that you did with FILE_NAME. It's listed after CMD_NAME, so any input that qualifies for both rules (like abc again) will hit CMD_NAME. Consider breaking these rules up into more conventional ones like so:

other_cmd : ID (ID | NUMBER)+ SEMI;   //instead of CMD_NAME ARG+
ID        : ('a'..'z'|'A'..'Z'|'_')+; //instead of CMD_NAME, "id" part of ARG
NUMBER    : ('0'..'9')+;              //"number" part of ARG
SEMI      : ';';

I added rule SEMI to mark the end of a command. Otherwise the parser won't know if input a b c d is supposed to be one command with three arguments (a(b,c,d)) or two commands with one argument each (a(b), c(d)).

Upvotes: 1

Related Questions