My simple ANTLR grammar ignores certain invalid tokens when parsing

Question

I asked a question a couple of weeks ago about my ANTLR grammar (My simple ANTLR grammar is not working as expected). Since asking that question, I've done more digging and debugging and gotten most of the kinks out. I am left with one issue, though.

My generated parser code is not picking up invalid tokens in one particular part of the text that is processed. The lexer is properly breaking things into tokens, but the parser does not kick out invalid tokens in some cases. In particular, when the invalid token is at the end of a phrase like "A and "B", the parser ignores it - it's like the token isn't even there.

Some specific examples:

• "A and B" - perfectly valid
• "A@ and B" - parser properly picks up the invalid @ token
• "A and @B" - parser properly picks up the invalid @ token
• "A and B@" - here's the mystery - the lexer finds the @ token and the parser IGNORES it (!)
• "(A and B@) or C" - further mystery - the lexer finds the @ token and the parser IGNORES it (!)

Here is my grammar:

grammar QvidianPlaybooks; 
options{ language=CSharp3; output=AST; ASTLabelType = CommonTree; } 

public parse
  :  expression
  ;

LPAREN : '(' ;
RPAREN : ')' ;
ANDOR : 'AND'|'and'|'OR'|'or';
NAME : ('A'..'Z');
WS : ' ' { $channel = Hidden; };
THEREST : .;

// ***************** parser rules:
expression : anexpression EOF!;
anexpression : atom (ANDOR^ atom)*;
atom : NAME | LPAREN! anexpression RPAREN!;

The code that then processes the resulting tree looks like this:

... from the main program
QvidianPlaybooksLexer lexer = new QvidianPlaybooksLexer(new ANTLRStringStream(src));
QvidianPlaybooksParser parser = new QvidianPlaybooksParser(new CommonTokenStream(lexer));
parser.TreeAdaptor = new CommonTreeAdaptor();

CommonTree tree = (CommonTree)parser.parse().Tree;
ValidateTree(tree, 0, iValidIdentifierCount);

// recursive code that walks the tree
public static RuleLogicValidationResult ValidateTree(ITree Tree, int depth, int conditionCount)
{
    RuleLogicValidationResult rlvr = null;

    if (Tree != null)
    {
        CommonErrorNode commonErrorNode = Tree as CommonErrorNode;
        if (null != commonErrorNode)
        {
            rlvr = new RuleLogicValidationResult();
            rlvr.IsValid = false;
            rlvr.ErrorType = LogicValidationErrorType.Other;
            Console.WriteLine(rlvr.ToString());
        }
        else
        {
            string strTree = Tree.ToString();
            strTree = strTree.Trim();
            strTree = strTree.ToUpper();
            if ((Tree.ChildCount != 0) && (Tree.ChildCount != 2))
            {
                rlvr = new RuleLogicValidationResult();
                rlvr.IsValid = false;
                rlvr.ErrorType = LogicValidationErrorType.Other;
                rlvr.InvalidIdentifier = strTree;
                rlvr.ErrorPosition = 0;
                Console.WriteLine(String.Format("CHILD COUNT of {0} = {1}", strTree, tree.ChildCount));
            }
            // if the current node is valid, then validate the two child nodes
            if (null == rlvr || rlvr.IsValid)
            {
                // output the tree node
                for (int i = 0; i < depth; i++)
                {
                    Console.Write("  ");
                }
                Console.WriteLine(Tree);
                rlvr = ValidateTree(Tree.GetChild(0), depth + 1, conditionCount);
                if (rlvr.IsValid)
                {
                    rlvr = ValidateTree(Tree.GetChild(1), depth + 1, conditionCount);
                }
            }
            else
            {
                Console.WriteLine(rlvr.ToString());
            }
        }
    }
    else
    {
        // this tree is null, return a "it's valid" result
        rlvr = new RuleLogicValidationResult();
        rlvr.ErrorType = LogicValidationErrorType.None;
        rlvr.IsValid = true;
    }

    return rlvr;
}

Answer 1

Add EOF to the end of your start rule. :)