CODEWITHSUNDEEP
c++parsingantlr4
Jason
Jason

Reputation: 551

Antlr4 grammar unexpected error (C++ target)

I'm using the C++ target of Antlr 4.7.1 and certain input strings trigger an unexpected error.

The string that doesn't parse with the grammar file as-is:

keys abc-def;

Error given:

Line(1:5) Error(mismatched input 'abc-def' expecting {KEY_NAME_PATTERN, STRING_LITERAL})

The strange thing is that if I swap the positions of the rules for KEY_NAME_IDENTIFIER and KEY_NAME_PATTERN in the grammar file then the string above parses ok, but the following string now fails (which passed without swapping the rule positions):

get key abc-def;

So I suspect a bug in Antlr, but am not 100% sure.

The grammar file:

grammar ModelKeyValue;

start : keyvalue_statements ;

keyvalue_statements: ( del_statement | get_statement | set_statement | keys_statement ) ';'
    ;

del_statement:
    KEYWORD_DEL key_name
    ;

get_statement:
    KEYWORD_GET key_name
    ;

set_statement:
    KEYWORD_SET key_name literal_value
    ;

keys_statement:
    KEYWORD_KEYS key_name_pattern
    ;

keyword:
    KEYWORD_DEL
    | KEYWORD_GET
    | KEYWORD_SET
    | KEYWORD_KEYS
    ;

key_name:
    KEY_NAME_IDENTIFIER
    | STRING_LITERAL
    ;

key_name_pattern:
    KEY_NAME_PATTERN
    | STRING_LITERAL
    ;

literal_value:
    STRING_LITERAL
    ;

KEYWORD_DEL: D E L;
KEYWORD_GET: G E T;
KEYWORD_SET: S E T;
KEYWORD_KEYS: K E Y S;

KEY_NAME_IDENTIFIER: ([a-z]|[A-Z]|[0-9]|'!'|'@'|'#'|'$'|':'|'-')+;

KEY_NAME_PATTERN: ([a-z]|[A-Z]|[0-9]|'!'|'@'|'#'|'$'|':'|'-'|'_'|'%')+;

STRING_LITERAL:
    '\'' ( ~'\'' | '\'\'' )* '\''
    | '"' ('\\' ~('\r' | '\n') | ~('\\' | '"'| '\r' | '\n'))* '"'
    ;

fragment A: [aA];
fragment B: [bB];
fragment C: [cC];
fragment D: [dD];
fragment E: [eE];
fragment F: [fF];
fragment G: [gG];
fragment H: [hH];
fragment I: [iI];
fragment J: [jJ];
fragment K: [kK];
fragment L: [lL];
fragment M: [mM];
fragment N: [nN];
fragment O: [oO];
fragment P: [pP];
fragment Q: [qQ];
fragment R: [rR];
fragment S: [sS];
fragment T: [tT];
fragment U: [uU];
fragment V: [vV];
fragment W: [wW];
fragment X: [xX];
fragment Y: [yY];
fragment Z: [zZ];

WS: [ \t\r\n]+ -> skip ;

If someone could verify this problem or suggest a fix that would help a lot, thanks.

Upvotes: 1

Views: 65

Answers (1)

Bart Kiers
Bart Kiers

Reputation: 170158

No, it is not a bug. You misunderstand how the tokenisation works. It's like this: whenever there are 2 (or more) rules that match the same amount of characters, like KEY_NAME_IDENTIFIER and KEY_NAME_PATTERN for the input keys abc-def;, the rule defined first "wins". The lexer does not "listen" to the parser (like a scanner-less parser works). There is no way around this.

To fix it, simply include KEY_NAME_IDENTIFIER in your key_name_pattern production:

key_name_pattern
 : KEY_NAME_IDENTIFIER
 | KEY_NAME_PATTERN
 | STRING_LITERAL
 ;

Upvotes: 1

Related Questions