CODEWITHSUNDEEP
javapythonantlr4
Bananasmoothii
Bananasmoothii

Reputation: 170

Antlr4/Java : how to make a semantic predicate that skips a token (lexer) according to the parser rule that calls it

I would like to use my lexer rule

NEW_LINE : '\n' -> skip;

Like a normal rule. Understanding by this: I want to ignore the new lines except when they are mandatory, to create a Python similar syntax. For example, here, new lines are ignored:

cook("banana",
     "potatoe)

but it is impossible to skip the new line for a new statement, like this:

cook("banana", "potatoe") varA = 12.4

, there must be a new line between cook() and the assignment. This is why I sometimes have to skip the new lines, but still force them somewhere else.

This is why I got this idea:

start
    : line*
    ;

line
    : line_expression (NEW_LINE | EOF)
    ;

line_expression
    : expression
    | assignment
    ;

expression
    : Decimal
    | Integer
    | Text
    | Boolean
    ;

And make a semantic predicate like "if the calling parser rule is not line, skip(); it." Now I just need help to do that.

I hope I was clear !

PS: I'm using Java as main language if that wasn't clear

Upvotes: 1

Views: 403

Answers (1)

Bart Kiers
Bart Kiers

Reputation: 170268

You could keep track of the number of ( you encounter (and decrease this numbers if you encounter a )). Then you only create NL tokens if this number is equal to zero.

Here's a quick demo:

grammar T;

@lexer::members {
  int parensLevel = 0;
}

parse
 : .*? EOF
 ;

OPAR    : '(' {parensLevel++;};
CPAR    : ')' {parensLevel--;};
NUMBER  : [0-9]+ ( '.' [0-9]+)?;
STRING  : '"' ~'"'* '"';
ASSIGN  : '=';
COMMA   : ',';
ID      : [a-zA-Z]+;
SPACES  : [ \t]+ -> skip;
NL      : {parensLevel == 0}? [\r\n]+;
NL_SKIP : [\r\n]+ -> skip;

If you feed the lexer the following input:

cook("banana",
     "potatoe")
  varA = 12.4

the following tokens will be created:

ID                        `cook`
'('                       `(`
STRING                    `"banana"`
','                       `,`
STRING                    `"potatoe"`
')'                       `)`
NL                        `\n`
ID                        `varA`
'='                       `=`
NUMBER                    `12.4`

As you can see, the NL inside the parens is skipped, while the one after the ) is not.

Upvotes: 1

Related Questions