CODEWITHSUNDEEP
javaantlr
Jason
Jason

Reputation: 4144

ANTLR3 Grammar Eating Integers

First, I'll admit: this is probably one of the ugliest grammars you will ever see in your life (ironically named SimplePolicy.g). It STARTED as a simple grammar, but I ended up needing more detailed exceptions, so I ended up making multiple exceptions extending RecognitionException. Then a RuntimeException to wrap it so I could get out of the parser methods.

This grammar is supposed to validate a group of one or more IF/THEN/ELSE?/ENDIF statements. Unfortunately, when I try to compare one of the INTEGER_FIELD tokens to a INTEGER token, the INTEGER token has totally disappeared from the TokenInputStream. I've tried sprinkling println()s through to figure out what is going on, with no luck.

Here is an example of a failure:

IF AGE < 21 THEN WT > 150 ENDIF

What seems to be happening is that the input String is good, but when I feed it to the Lexer via a CharStream, the integer constant disappears that AGE and WT are being compared to disappear.

Here is a unit test:

@Test
public void ifStatement_legalPolicy2_passes() {
    String policy = "IF AGE < 21 THEN "+ 
                    "WT > 150 "+ 
                    "ENDIF";


    CharStream charStream = new ANTLRStringStream(policy);
    SimplePolicyLexer lexer = new SimplePolicyLexer(charStream);
    TokenStream tokenStream = new CommonTokenStream(lexer);
    System.out.println(tokenStream.toString());
    SimplePolicyParser parser = new SimplePolicyParser(tokenStream);    

    try {
        parser.ifStatement();
        parser.reset();
    } catch (RecognitionRuntimeException e) {
        RecognitionException e1 = e.getRecognitionException();
        fail("Legal policy should not have thrown exception. Exception thrown: " + e1.getClass().getName() + ". Line: " + e1.line + ", Column: " + e1.charPositionInLine);
    } catch (RecognitionException e) {
    }       


}

Here is the String output of the tokenstream:

13:14:14.978 DEBUG m.a.k.m.impl.SimplePolicyParser - ifStatement: IF AGE <  THEN WT >  ENDIF

I apologize for the uglyness of this grammar, but if anyone can tell me why this dies I would be greatly appreciative.

grammar SimplePolicy;

options {
  language = Java;
  backtrack = true;
}


@header {
package com.manager.impl;

import com.manager.RecognitionRuntimeException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Arrays;


@lexer::header {
package com.manager.impl;

}

@lexer::members {

}

@parser::members {

 private static final Logger log = LoggerFactory.getLogger(SimplePolicyParser.class);

 @Override
 protected Object recoverFromMismatchedToken(IntStream input, int ttype, BitSet follow) throws RecognitionException {
   throw new MismatchedTokenException(ttype, input);
 }

  @Override
 public Object recoverFromMismatchedSet(IntStream input, RecognitionException e, BitSet follow) throws RecognitionException {
   throw e;
 }

 @Override
 public String getErrorMessage(RecognitionException e, String[] tokenNames) {
  // wrap in a runtime exception to escape ANTLR's dungeon
  throw new RecognitionRuntimeException(e);
 }

}

@rulecatch {
    catch (RecognitionException e) {
           System.out.println(getErrorHeader(e));
      //System.out.println(getErrorMessage(e,tokenNames));
        throw new RecognitionRuntimeException(e);
    }
}

// evaluate multiple show statements
policyGroupWithShow
  : show (show)* EOF
  | ifStatement+ EOF
  {
    // nope, this isn't legal
    ShowExpectedException ex = new ShowExpectedException();
    ex.line = 1;
    ex.charPositionInLine=0;
    throw ex;
  }
  ;

policyGroupWithoutShow
  : ifStatement (ifStatement)* EOF
  | show+ EOF
  {
    // not legal
    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = 1;
    ex.charPositionInLine = 0;
    throw ex;
  }
  ;

//policyGroup
//  : show (show)* EOF
//  ;


// evaluate a single SHOW statement
show
  //: ifStatement
  : SHOW STRING FOR ifStatement // good
  // missing for (FOR expected)
  | SHOW expr1a=STRING ifStatement // missing for
  {
    int nextTokenPosition = expr1a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(nextTokenPosition + 1);

    ShowWithoutForException ex = new ShowWithoutForException();
    ex.line = expr1a.getLine();
    ex.charPositionInLine = token.getCharPositionInLine();
    throw ex;

  }
  // missing show (SHOW expected)
  | expr1b=STRING FOR ifStatement
  {
    int tokenPosition = expr1b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition);
    ForWithoutShowException ex = new ForWithoutShowException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  ;


ifStatement
  @init {
    log.debug("ifStatement: " + input.toString());
  }
  //: IF logical THEN logical+ (ELSE logical+)? ENDIF
  : IF operation 'THEN' expression+ ENDIF // good
  {
    log.debug("Matched [IF logical THEN expression+ ENDIF]");
  }
  | IF logical THEN expression+ ELSE expression+ ENDIF
  {
    log.debug("Matched [IF logical THEN expression+ ELSE expression+ ENDIF]");
  }
  | logical expr1a=THEN expression* (ELSE expression*)? ENDIF // missing if
  {
    int tokenPosition = expr1a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition);
    MissingIfException ex = new MissingIfException();
    ex.line = token.getLine();
    ex.charPositionInLine = 0;
    throw ex;
  }
  // missing THEN (THEN clause is missing)
  | expr1b=IF logical expression+ (ELSE expression+)? ENDIF
  {
    int tokenPosition = expr1b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
    MissingThenClauseException ex = new MissingThenClauseException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getCharPositionInLine();
    throw ex;
  }
  // missing ELSE or ENDIF (ENDIF Expected)
  | IF logical expr1c=THEN expression+
  {
    String inputText = input.toString();
    if (inputText.indexOf("ELSE") < 0 && inputText.indexOf("ENDIF") < 0) {
      // best I can do is get the line/column from the THEN keyword
        MissingElseOrEndifException ex = new MissingElseOrEndifException();
        ex.line = expr1c.getLine();
        ex.charPositionInLine = expr1c.getCharPositionInLine();
        throw ex;   
    }
  }
  // missing comparison for IF (rule expected)
  | IF expr1d=THEN expression* ENDIF
  {
    int tokenPosition = expr1d.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition - 1);

    RuleExpectedException ex = new RuleExpectedException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStopIndex();
    throw ex;
  }
  // missing body of then (unexpected symbol or construction)
  | IF logical a=THEN b=ENDIF
  {
    int tokenPosition1 = a.getTokenIndex();
    int tokenPosition2 = b.getTokenIndex();
    CommonToken tokenA = (CommonToken) input.get(tokenPosition1);
    CommonToken tokenB = (CommonToken) input.get(tokenPosition2);

    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    //MissingThenBodyException ex = new MissingThenBodyException();
    if (tokenA.getLine() == tokenB.getLine()) {
      ex.line = tokenA.getLine();
      ex.charPositionInLine = tokenA.getCharPositionInLine();
    } else {
      ex.line = tokenA.getLine() + 1;
      ex.charPositionInLine = 0;
    }
    throw ex;
  }
  // missing body of ELSE (Unexpected symbol or construction)
  | IF logical THEN expression+ ELSE expr3e=ENDIF
  {

    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = expr3e.getLine();
    ex.charPositionInLine = expr3e.getCharPositionInLine();
    throw ex;
  }
  // body of IF missing (Body of IF clause is missing)
  | IF expr3f=ENDIF
  {
    int tokenPosition = expr3f.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition);

    MissingIfBodyException ex = new MissingIfBodyException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getCharPositionInLine();
    throw ex;
  }
  | expression // expression should just pass through
  ;


expression
  @init {
    log.debug("Expression: " + input.toString());
  }
  : logical
  ;

// deal with AND/OR operator
logical
  @init {
    log.debug("Logical:" + input.toString());
  }
 : operation (AND operation | OR operation)*
 ;

operation
  @init {
    log.debug("Operation:" + input.LT(1) + input.LT(2) + input.LT(3));
  }
  // good rules
  //: ';' ~('/r/n'|EOF)*
  : DATE_FIELD (EQ|NE|LT|LE|GT|GE) (DATE_FIELD|DATE|DATE_CONSTANT)
  {

    log.info("Matched STRING_FIELD (EQ|NE) (STRING_FIELD|STRING) field operation: " + input.toString());
  }
  | DATE_FIELD (PLUS|MINUS) (DATE_FIELD|DATE|DATE_CONSTANT|DATE_PERIOD_FIELD|datePeriod)
  {
    log.info("Matched DATE_FIELD (PLUS|MINUS) (DATE_FIELD|DATE|DATE_CONSTANT|DATE_PERIOD_FIELD|datePeriod) field operation: " + input.toString());
  }
  | DATE_PERIOD_FIELD (EQ|NE|LT|LE|GT|GE) (DATE_PERIOD_FIELD|DATE_PERIOD_CONSTANT)
  {
    log.info("Matched DATE_PERIOD_FIELD (EQ|NE|LT|LE|GT|GE) (DATE_PERIOD_FIELD|DATE_PERIOD_CONSTANT) field operation: " + input.toString());
  }
  | DATE_PERIOD_FIELD (PLUS|MINUS) (DATE_PERIOD_FIELD|datePeriod|DATE_FIELD|DATE_CONSTANT|DATE)
  {
    log.info("Matched DATE_PERIOD_FIELD (PLUS|MINUS) (DATE_PERIOD_FIELD|datePeriod|DATE_FIELD|DATE_CONSTANT|DATE) field operation: " + input.toString());
  }
  | STRING_FIELD (EQ|NE) (STRING_FIELD|STRING)
  {
  log.info("Matched STRING_FIELD (EQ|NE) (STRING_FIELD|STRING) field operation: " + input.toString());
  }
  | INTEGER_FIELD (EQ|NE|LT|LE|GT|GE) test=inAllFields
  {
    Boolean result = $inAllFields.result;
    Integer type = $inAllFields.tokenType;

    //int testType = test.getType();
    if (type != INTEGER && type != INTEGER_FIELD) {

        IncompatibleTypeException ex = new IncompatibleTypeException();
        //ex.line = test.getLine();
        //ex.charPositionInLine = test.getStartIndex();
        throw ex;
    }
  }
  | INTEGER_FIELD (EQ|NE|LT|LE|GT|GE) test1=(INTEGER_FIELD|INTEGER)
  {
    log.info(test1.getText());
    log.info("Matched INTEGER_FIELD (EQ|NE|LT|LE|GT|GE) (INTEGER_FIELD|INTEGER) field operation: " + input.toString());
  }
  | BOOLEAN_FIELD (EQ|NE) (BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
  log.info("Matched BOOLEAN_FIELD (EQ|NE) (BOOLEAN_FIELD|BOOLEAN_CONSTANT) field operation: " + input.toString());
  }
  | COMMENT
  {
  log.info("Matched COMMENT field operation: " + input.toString());

  }
  // specify bad rules
  // defined fields with no operation. The expression will be null if datePeriod isn't split off
  | (INTEGER_FIELD|INTEGER) (PLUS|MINUS|EQ|NE|LT|LE|)
  {
    System.out.println("Will it work?");
  }
  | (datePeriod)|e1=(DATE_PERIOD_FIELD|DATE_FIELD|STRING_FIELD|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    RelationalOperatorOrNotExpectedException ex = new RelationalOperatorOrNotExpectedException();
    if (e1 == null) {
        ex.line = 1;
        ex.charPositionInLine = 0;
    } else {
      ex.line = e1.getLine();
      ex.charPositionInLine = e1.getCharPositionInLine();
    }
    throw ex;
  }
  | e2=(INTEGER|DATE|DATE_CONSTANT)
  {
    RuleCannotBeginWithNumberOrDateException ex = new RuleCannotBeginWithNumberOrDateException();
    ex.line = e2.getLine();
    ex.charPositionInLine = e2.getCharPositionInLine();
    throw ex;
  }
  // attempt to compare a date field to a different field type. Incompatable Type exception
  | DATE_FIELD operator1a=(EQ|NE|LT|LE|GT|GE) (DATE_PERIOD_FIELD|datePeriod|STRING_FIELD|STRING|INTEGER_FIELD|INTEGER|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator1a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 2);

    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  // attempt to add/sub illegal field types to date (Relational Operator or NOT expected)
  | DATE_FIELD operator1b=(PLUS|MINUS)
      (STRING_FIELD|STRING|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator1b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition+1);

    RelationalOperatorOrNotExpectedException ex = new RelationalOperatorOrNotExpectedException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex; 
  }
  // attempting to add an int to a date field makes it assume it's a bad date period
  | DATE_FIELD operator1c=(PLUS|MINUS) expr1c=(INTEGER)
  {
   int tokenPosition = operator1c.getTokenIndex();
   CommonToken token = (CommonToken) input.get(tokenPosition + 1);

   MissingYearsMonthsWeeksDaysException ex = new MissingYearsMonthsWeeksDaysException();
   ex.line = token.getLine();
   ex.charPositionInLine = token.getStartIndex();
   throw ex;
  }
  // attempt to compare a String field to something besides a String field or String constant
  | STRING_FIELD operator2a=(EQ|NE) ~(STRING_FIELD|STRING)
  {
    int tokenPosition = operator2a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | expr2b=STRING_FIELD operator2b=(LT|LE|GT|GE) (STRING_FIELD|STRING)
  {
    int tokenPosition = operator2b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | stringExpr3=STRING
  {
    RuleCannotBeginWithAStringException ex = new RuleCannotBeginWithAStringException();
    ex.line = stringExpr3.getLine();
    ex.charPositionInLine = stringExpr3.getCharPositionInLine();
    throw ex;
  }
  // attempt to compare a number field to something besides a num field or int constant

  | INTEGER_FIELD operator3a=(EQ|NE|LT|LE|GT|GE) expr3a2=(DATE_FIELD|DATE|STRING_FIELD|STRING|DATE_PERIOD_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    Boolean value = $inAllFields.result;
    //System.out.println("Expr text: " + expr3a2.getText());
    int tokenPosition = operator3a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  // attempt to perform a PLUS/MINUS on a number field with something besides a num field or constant int
  | INTEGER_FIELD operator3b=(PLUS|MINUS)
    (DATE_FIELD|DATE|STRING_FIELD|STRING|DATE_PERIOD_FIELD|datePeriod|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator3b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex; 
  }
  // attempt to compare date period to something besides a date period field or date period constant
  | DATE_PERIOD_FIELD operator4a=(EQ|NE|LT|LE|GT|GE)
    (INTEGER_FIELD|INTEGER|DATE_FIELD|DATE|DATE_CONSTANT|STRING_FIELD|STRING|BOOLEAN_FIELD|BOOLEAN_CONSTANT|WAIVER_FIELD)
  {
    int tokenPosition = operator4a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | DATE_PERIOD_FIELD operator5a=(PLUS|MINUS) (STRING_FIELD|STRING|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator5a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    RelationalOperatorOrNotExpectedException ex = new RelationalOperatorOrNotExpectedException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | DATE_PERIOD_FIELD operator5b=(PLUS|MINUS) INTEGER
  {
    int tokenPosition = operator5b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    MissingYearsMonthsWeeksDaysException ex = new MissingYearsMonthsWeeksDaysException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | WAIVER_FIELD operator6a=(HAS|EQ|NE) ~(STRING)
  {
    int tokenPosition = operator6a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | WAIVER_FIELD operator6b=(PLUS|MINUS|LT|LE|GT|GE)  STRING
  {
    int tokenPosition = operator6b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | BOOLEAN_FIELD operator7a=(EQ|NE) ~(BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator7a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex; 
  }
  | BOOLEAN_FIELD operator7b=(PLUS|MINUS|LT|LE|GT|GE) (DATE_FIELD|DATE|DATE_CONSTANT|DATE_PERIOD_FIELD|datePeriod|STRING_FIELD|STRING|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator6b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);

    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  ;

// this rule checks that a value is valid. It's not in the validation
// chain
atom
  : datePeriod
  | DATE_FIELD
  | INTEGER_FIELD
  | STRING_FIELD
  | DATE_PERIOD_FIELD
  | WAIVER_FIELD
  | BOOLEAN_FIELD
  | INTEGER
  {
    System.out.println("Matched atom INTEGER");
  }
  | DATE
  | DATE_CONSTANT
  | BOOLEAN_CONSTANT
  | STRING
  //| LPAREN expression RPAREN
  ;

datePeriod
  : (DATE_PERIOD_CONSTANT)+
  ;

// "subatomic" rules, meant to feed the atom rule understandable values



fragment DIGIT: ('0'..'9');

DATE_FIELD:('DOB'|'TEST_DATE');
DATE_PERIOD_FIELD:('AFS');
BOOLEAN_FIELD:('CERTIFIED'|'OVERRIDE');
INTEGER_FIELD:('AGE'|'HT'|'WT');
STRING_FIELD:('CURR_LOC'|'LANG_1'|'LANG_2''USER_LEVEL');
WAIVER_FIELD:('WAIVER');

DATE: /* empty */ {System.out.println("Matched DATE");};

INTEGER: DIGIT+
//INTEGER:('0'..'9')+
{
  System.out.println("Matched INTEGER: " + $text);
  //$type = INTEGER;
    // dynamically change the type when we match the date regular expression
  if ($text.matches("(19|20|21)[0-9]{2}[0-1]\\d{3}")) {
  //if ($text.matches("(19|20|21)[0-9]{2}[0-1]\\d{3}")) {
    System.out.println("Matched date pattern");
    $type = DATE;
  }

}
;
DATE_PERIOD_CONSTANT: ((INTEGER ' ' YEAR)|(INTEGER ' ' MONTH)|(INTEGER ' ' WEEK)|(INTEGER ' ' DAY))
{
  System.out.println("Matched DATE_PERIOD_CONSTANT");
};
DATE_CONSTANT:('TODAY'|'YESTERDAY'|'TOMMOROW');
BOOLEAN_CONSTANT:('TRUE'|'FALSE'|'"Y"'|'"N"');
IF: 'IF';
THEN: 'THEN';
ELSE: 'ELSE';
ENDIF: 'ENDIF';
AND: 'AND';
OR: 'OR';
YEAR: ('YEAR'|'YEARS');
MONTH: ('MONTH'|'MONTHS');
WEEK: ('WEEK'|'WEEKS');
DAY: ('DAY'|'DAYS');

STRING: '"' ID  (' ' ID)* '"' {System.out.println("Matched STRING");};
//  {
//    // strip the quotes once we match this token
//    setText(getText().substring(1, getText().length()-1));
//  }
//  ;

EQ: '=' {System.out.println("Matched EQ");};
NE: '<>';
LT: '<';
LE: '<=';
GT: '>';
GE: '>=';
HAS: 'HAS';
LPAREN: '(';
RPAREN: ')';
PLUS: '+';
MINUS: '-';
SHOW: 'SHOW';
FOR: 'FOR';
ID: ('A'..'Z'|'a'..'z'|'0'..'9'|','|'!'|'?'|':')+ {System.out.println("Matched ID: " + $text);};

COMMENT: ';' ~('\r'|'\n')* {skip();};
WS: (' '+|'\r'|'\n'|'\t') {$channel = HIDDEN;};

Upvotes: 2

Views: 174

Answers (1)

Bart Kiers
Bart Kiers

Reputation: 170227

Whenever your lexer "sees" a number followed by a space, it tries to construct a DATE_PERIOD_CONSTANT. Make that a parser rule instead:

date_persiod_constant
 : INTEGER (YEAR | MONTH | WEEK | DAY)
 ;

(and remove DATE_PERIOD_CONSTANT, of course)

Also, your DATE rule matches nothing. I understand you need this rule to change the type of your INTEGER rule, but never ever create lexer rules that (possibly) match nothing. Make DATE a fragment instead:

fragment DATE : ;

Even though it is a fragment you can then still use DATE in your parser rules. By making it a fragment, you're just instructing the lexer to never create DATE tokens on its own, but you know the lexer can produce such tokens since you're changing the type of INTEGER when it looks like a date.

EDIT

Is there any way to keep it as a lexer rule? I would really like the date constant to emit a token that I can use later.

Sure, it's possible. Not pretty, but possible:

INTEGER
 : DIGIT+ ( (' '+ (YEAR | MONTH | WEEK | DAY))=> 
             ' '+ (YEAR | MONTH | WEEK | DAY) {$type = DATE_PERIOD_CONSTANT;}
          )?
   {
     if ($text.matches("(19|20|21)[0-9]{2}[0-1]\\d{3}")) {
       $type = DATE;
     }
   }

 ;

fragment DATE_PERIOD_CONSTANT : ;

What this does is after matching one or more digits, DIGIT+, it forces the lexer to look ahead in the character stream to see if there are one or more spaces followed by a YEAR, MONTH, WEEK or DAY, the predicate (' '+ (YEAR | MONTH | WEEK | DAY))=>, and if so, it goes ahead and matches this (and you change the type from INTEGER to DATE_PERIOD_CONSTANT).

EDIT II

One final issue: a DATE period can be something like "1 YEAR 4 MONTHS 2 DAYS", not just a single such as "1 YEAR". Attempting to add a + to the rule was not successful. Can I ask for one more assist in order to make that rule work the way I need it to?

Something like this should do it (but again, a parser rule might be more appropriate!):

INTEGER
 : (DATE_PERIOD_CONSTANT)=> DATE_PERIOD_CONSTANT ((' '+ DATE_PERIOD_CONSTANT)=> ' '+ DATE_PERIOD_CONSTANT)*
   {
     $type=DATE_PERIOD_CONSTANT;
   }
 | DIGIT+
   {
     if ($text.matches("(19|20|21)[0-9]{2}[0-1]\\d{3}")) {
       System.out.println("Matched date pattern");
       $type = DATE;
     }
   }   
 ;

fragment DATE_PERIOD_CONSTANT
 : DIGIT+ ' '+ (YEAR | MONTH | WEEK | DAY)
 ;

Upvotes: 4

Related Questions