CODEWITHSUNDEEP
commentsantlr4lexer
Terry151151
Terry151151

Reputation: 771

ANTLR4. Comments not being swallowed by the lexer

I have the following Antlr4 grammar:

grammar Smarty;

/*
 * Parser Rules
 */

parse: smartyBody
     ;

smartyBody: include
          | bodyText?
          ;

include: INCLUDE fileName CB
          ;

bodyText: BODY_TEXT
         ;

fileName: FILENAME
        ; 
/*
 * Lexer Rules
 */
COMMENT: '{*' .*? '*}'      -> skip;

INCLUDE: '{include' SPACE+ 'file='?;

BODY_TEXT: ANY_CHAR+;

CB: SPACE? '}';

FILENAME: '\'' FILE_NAME '\''
     |    '"' FILE_NAME '"';

SPACE: [ \t];
NEW_LINE:   [\r\n]+         -> skip;

fragment FILE_NAME: PATH_CHARACTERS+;
fragment PATH_CHARACTERS: ~[\u0000-\u001f"*:<>?\\/|\u007f-\uffff];
fragment ANY_CHAR: [ -~];

Now if i try to use comments they are passed back and not swallowed.

Text passed to the parser: {* {include file='xxxxx'} *} zzzzzzzz

If I override EnterBodyText and pass the text string to the parser the Text variable in the listener get the original text including the comments.

internal partial class BaseListener : SmartyBaseListener
{
    public override void EnterBodyText( SmartyParser.BodyTextContext context )
    {
       var Text = context.BODY_TEXT().GetText();
       // Text gets: {* {include file='xxxxx'} *} zzzzzzzz
       // not just zzzzzzzz
    }
}

Any help appreciated.

Upvotes: 2

Views: 326

Answers (1)

Bart Kiers
Bart Kiers

Reputation: 170308

ANTLR's lexer will try to match as much as possible. So the input {* {include file='xxxxx'} *} zzzzzzzz is matched entirely by BODY_TEXT.

One way to solve this is make BODY_TEXT a parser rule instead:

body_text: ANY_CHAR+;

...

COMMENT: '{*' .*? '*}' -> skip;

...

// No fragment anymore. And keep this as the last lexer rule!
ANY_CHAR: [ -~];

Upvotes: 2

Related Questions