antlr grammar for comments, and plain text between tags

Question

Follow-up question to : Getting plain text in antlr instead of tokens

1.I used a rule

COMMENT : START_1_TAG START_COMMENT END_1_TAG .*? START_2_TAG END_COMMENT END_2_TAG -> skip;

to skip any comments using my lexer. But, I get a mismatched input , when I give any space inside the tags.

The part of my relevant part of my Lexer is:

lexer grammar DemoLexer;

START_1_TAG : '<%' -> pushMode(IN_TAG);
START_2_TAG : '<<' -> pushMode(IN_TAG);

COMMENT : START_1_TAG START_COMMENT END_1_TAG .*? START_2_TAG END_COMMENT END_2_TAG  -> skip;

TEXT        : ( ~[<] | '<' ~[<%] )+;

mode IN_TAG;
START_COMMENT : 'startcomment' ;
END_COMMENT : 'endcomment' ;
ID         : [A-Za-z_][A-Za-z0-9_]*;
INT_NUMBER : [0-9]+;
END_1_TAG  : '%>' -> popMode;
END_2_TAG  : '>>' -> popMode;
SPACE      : [ \t\r\n] -> channel(HIDDEN);

My issue is, <%comment%>hi<%endcomment%> gets parsed correctly. But, while I give my input as, <% comment %> or <% endcomment %>, with spaces in between the tags, it is not recognized by the COMMENT rule.

It gets recognized by the COMMENT rule, when I define the rule as:

COMMENT : START_1_TAG SPACE*? 'commentstart' SPACE*? END_1_TAG .*? START_1_TAG SPACE*? 'commentend' SPACE*? END_1_TAG -> skip;

with explicit spaces.

Is this the proper method to handle this?

2.I have a rule where I need the raw content inside a tag pair. Eg:

Here, the tokens need to be <%startraw%>,<%Hi%> and <%endraw%>

I tried using the text rule, but it dosen't work because it dosen't include '<%' and '<<'.

I tried:

in my parser,

rawText : RAW_TAG_START RAW_TEXT RAW_TAG_END ;

in my lexer,

RAW_TAG_START : '<%' 'startraw' '%>' -> pushMode(RAW_MODE);
RAW_TAG_END : '<%' 'endraw' '%>' -> popMode; 
mode RAW_MODE;
  RAW_TEXT : .*? ;

For some reason, when I try to parse this with the intellij antlr plugin, it seems to freeze and crash whenever I try to match the rawText rule.

Answer 1

Is this the proper method to handle this?

No, I 'd say it's not. In this case, it's not just a plain comment, but it's a regular tag that happens to represent a comment. Therefor I'd treat it as any other tag (define it in the parser, not the lexer).

For some reason, when I try to parse this with the intellij antlr plugin, it seems to freeze and crash whenever I try to match the rawText rule.

That could be because of this: RAW_TEXT : .*? ; that matches an empty string, and causes the lexer to produce infinite amount of tokens.

I would do something like this:

lexer grammar DemoLexer;

START_1_TAG : '<%' -> pushMode(IN_TAG);
START_2_TAG : '<<' -> pushMode(IN_TAG);
TEXT        : ( ~[<] | '<' ~[<%] )+;

fragment S  : [ \t\r\n];
fragment ID : [A-Za-z_][A-Za-z0-9_]*;

mode IN_TAG;
  START_RAW     : 'raw' S* '%>' -> pushMode(IN_RAW);
  START_COMMENT : 'comment';
  END_COMMENT   : 'endcomment';
  END_ID        : 'end' ID;
  START_ID      : ID;
  INT_NUMBER    : [0-9]+;
  END_1_TAG     : '%>' -> popMode;
  END_2_TAG     : '>>' -> popMode;
  SPACE         : [ \t\r\n] -> channel(HIDDEN);

mode IN_RAW;
  END_RAW : '<%' S* 'endraw' S* '%>' -> popMode, popMode; // pop twice: out of IN_RAW and IN_TAG!
  ANY_RAW : . ; // No '+' or '*', just a single token!

A demo parser:

parser grammar DemoParser;

options {
  tokenVocab=DemoLexer;
}

code
 : codeBlock* EOF
 ;

codeBlock
 : TEXT
 | tag1Ops
 | tag2Ops
 ;

tag1Ops
 : rawTag
 | commentTag
 | otherTag
 ;

rawTag
 : START_1_TAG START_RAW ANY_RAW* END_RAW
 ;

commentTag
 : START_1_TAG START_COMMENT END_1_TAG TEXT START_1_TAG END_COMMENT END_1_TAG
 ;

otherTag
 : START_1_TAG START_ID END_1_TAG TEXT START_1_TAG END_ID END_1_TAG
 ;

tag2Ops
 : START_2_TAG START_ID END_2_TAG TEXT START_2_TAG END_ID END_2_TAG
 ;

And a little main class to test it all:

import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.ParseTree;

public class Main {

  public static void main(String[] args) {

    String source = "aaa <% raw %> RAW <% endraw %> " +
            "bbb " +
            "<% foo %> FOO <% endfoo %> " +
            "ccc " +
            "<%comment%> COMMENT <%endcomment%> " +
            "ddd";

    DemoLexer lexer = new DemoLexer(CharStreams.fromString(source));
    DemoParser parser = new DemoParser(new CommonTokenStream(lexer));
    ParseTree tree = parser.code();

    System.out.println(tree.toStringTree(parser));
  }
}

which will print:

(code
  (codeBlock aaa )
  (codeBlock (tag1Ops (rawTag <% raw %>   R A W   <% endraw %>)))
  (codeBlock  bbb )
  (codeBlock (tag1Ops (otherTag <% foo %>  FOO  <% endfoo %>)))
  (codeBlock  ccc )
  (codeBlock (tag1Ops (commentTag <% comment %>  COMMENT  <% endcomment %>)))
  (codeBlock  ddd)
  <EOF>)

(I added some manual line breaks for clarity)

And the ANTLR IntelliJ plugin can also cope with it:

EDIT

If you're set on skipping comments in the lexer (which I wouldn't do), then you could do something like this:

lexer grammar DemoLexer;

COMMENT     : '<%' S* 'comment' S* '%>' .*? '<%' S* 'endcomment' S* '%>' -> skip;
START_1_TAG : '<%' -> pushMode(IN_TAG);
START_2_TAG : '<<' -> pushMode(IN_TAG);
TEXT        : ( ~[<] | '<' ~[<%] )+;

EDIT II

If you've measured that ANY_RAW has a significant impact on performance, you could do something like this:

mode IN_RAW;
  END_RAW  : '<%' S* 'endraw' S* '%>' -> popMode, popMode;
  SAFE_RAW : ( ~[<] | '<' ~[<%] )+

  // Fall through to match "<" from "<% ..." that are not matched by END_RAW
  OTHER_RAW  : . ;