How can I parse a special character differently in two terminal rules using antlr?

Question

I have a grammar that uses the $ character at the start of many terminal rules, such as $video{, $audio{, $image{, $link{ and others that are like this.

However, I'd also like to match all the $ and { and } characters that don't match these rules too. For example, my grammar does not properly match $100 in the CHUNK rule, but adding the $ to the long list of acceptable characters in CHUNK causes the other production rules to break.

How can I change my grammar so that it's smart enough to distinguish normal $, { and } characters from my special production rules?

Basically what I'd to be able to do is say, "if the $ character doesn't have {, video, image, audio, link, etc. after it, then it should go to CHUNK".

grammar Text;

@header {
}

@lexer::members {
    private boolean readLabel = false;
    private boolean readUrl = false;
}

@members {
    private int numberOfVideos = 0;
    private int numberOfAudios = 0;
    private StringBuilder builder = new StringBuilder();

    public String getResult() {
        return builder.toString();
    }
}

text
    :   expression*
    ;

expression
    :   fillInTheBlank 
        {
            builder.append($fillInTheBlank.value);
        }
    |   image 
        {
            builder.append($image.value);
        }
    |   video
        {
            builder.append($video.value);
        }
    |   audio
        {
            builder.append($audio.value);
        }
    |   link
        {
            builder.append($link.value);
        }
    |   everythingElse
        {
            builder.append($everythingElse.value);
        }
    ;

fillInTheBlank returns [String value]
    :   BEGIN_INPUT LABEL END_COMMAND
        {
            $value = "<input type=\"text\" id=\"" +
                $LABEL.text +
                "\" name=\"" + 
                $LABEL.text +
                "\" class=\"FillInTheBlankAnswer\" />";
        }
    ;

image returns [String value]
    :   BEGIN_IMAGE URL END_COMMAND
        {
            $value = "<img src=\"" + $URL.text + "\" />";
        }
    ;

video returns [String value]
    :   BEGIN_VIDEO URL END_COMMAND
        {
            numberOfVideos++;

            StringBuilder b = new StringBuilder();
            b.append("<div id=\"video1\">Loading the player ...</div>\r\n");
            b.append("<script type=\"text/javascript\">\r\n");
            b.append("\tjwplayer(\"video" + numberOfVideos + "\").setup({\r\n");
            b.append("\t\tflashplayer: \"/trainingdividend/js/jwplayer/player.swf\", file: \"");
            b.append($URL.text);
            b.append("\"\r\n\t});\r\n");
            b.append("</script>\r\n");

            $value = b.toString();
        }
    ;

audio returns [String value]
    :   BEGIN_AUDIO URL END_COMMAND
        {
            numberOfAudios++;

            StringBuilder b = new StringBuilder();
            b.append("<p id=\"audioplayer_");
            b.append(numberOfAudios);
            b.append("\">Alternative content</p>\r\n");
            b.append("<script type=\"text/javascript\">\r\n");
            b.append("\tAudioPlayer.embed(\"audioplayer_");
            b.append(numberOfAudios);
            b.append("\", {soundFile: \"");
            b.append($URL.text);
            b.append("\"});\r\n");
            b.append("</script>\r\n");

            $value = b.toString();
        }
    ;   

link returns [String value]
    :   BEGIN_LINK URL END_COMMAND
        {
            $value = "<a href=\"" + $URL.text + "\">" + $URL.text + "</a>";
        }
    ;   

everythingElse returns [String value]
    :   CHUNK
        {
            $value = $CHUNK.text;
        }
    ;

BEGIN_INPUT
    :   '${' 
        { 
            readLabel = true; 
        }
    ;

BEGIN_IMAGE
    :   '$image{' 
        { 
            readUrl = true; 
        }
    ;

BEGIN_VIDEO
    :   '$video{' 
        { 
            readUrl = true; 
        }
    ;

BEGIN_AUDIO
    :   '$audio{' 
        { 
            readUrl = true; 
        }
    ;

BEGIN_LINK
    :   '$link{' 
        { 
            readUrl = true; 
        }
    ;

END_COMMAND
    :   { readLabel || readUrl }?=> '}' 
        { 
            readLabel = false; 
            readUrl = false;
        }
    ;

URL
    :   { readUrl }?=> 'http://' ('a'..'z'|'A'..'Z'|'0'..'9'|'.'|'/'|'-'|'_'|'%'|'&'|'?'|':')+
    ;

LABEL
    :   { readLabel }?=> ('a'..'z'|'A'..'Z') ('a'..'z'|'A'..'Z'|'0'..'9')*
    ;

CHUNK
    //: (~('${'|'$video{'|'$image{'|'$audio{'))+
    :   ('a'..'z'|'A'..'Z'|'0'..'9'|' '|'\t'|'\n'|'\r'|'-'|','|'.'|'?'|'\''|':'|'\"'|'>'|'<'|'/'|'_'|'='|';'|'('|')'|'&'|'!'|'#'|'%'|'*')+
    ;

Answer 1

You can't negate more than a single character. So, the following is invalid:

~('${')

But why not simply add '$', '{' and '}' to your CHUNK rule and remove the + at the end of the CHUNK rule (otherwise it would gobble up to much, possibly '$video{' further in the source, as you have noticed yourself already)?.

Now a CHUNK token will always consist of a single character, but you could create a production rule to fix this:

chunk
  :  CHUNK+
  ;

and use chunk in your production rules instead of CHUNK (or use CHUNK+, of course).

Input like "{ } $foo $video{" would be tokenized as follows:

CHUNK                {
CHUNK
CHUNK                }
CHUNK
CHUNK                $
CHUNK                f
CHUNK                o
CHUNK                o
CHUNK
BEGIN_VIDEO          $video{

EDIT

And if you let your parser output an AST, you can easily merge all the text that one or more CHUNK's match into a single AST, whose inner token is of type CHUNK, like this:

grammar Text;

options {
  output=AST;
}

...

chunk
  :  CHUNK+ -> {new CommonTree(new CommonToken(CHUNK, $text))}
  ;

...

Answer 2

An alternative solution which doesn't generate that many single-character tokens would be to allow chunks to contain a $ sign only as the first character. That way your input data will get split up at the dollar signs only.

You can achieve this by introducing a fragment lexer rule (i.e., a rule that does not define a token itself but can be used in other token regular expressions):

fragment CHUNKBODY
    : 'a'..'z'|'A'..'Z'|'0'..'9'|' '|'\t'|'\n'|'\r'|'-'|','|'.'|'?'|'\''|':'|'\"'|'>'|'<'|'/'|'_'|'='|';'|'('|')'|'&'|'!'|'#'|'%'|'*';  

The CHUNK rule then looks like:

CHUNK
    :   { !readLabel && !readUrl }?=> (CHUNKBODY|'$')CHUNKBODY*
    ;

This seems to work for me.