CODEWITHSUNDEEP
antlrantlr3lexer
slmyers
slmyers

Reputation: 767

ANTLR3 lexer returns one token when expecting to return 5 tokens

Hello i'm trying to build a simple lexer to tokenize lines starting with an ';' character.

This is my lexer grammar:

lexer grammar TestLex;

options {
  language = Java;
  filter = true;
}

@header {
  package com.ualberta.slmyers.cmput415.assign1;
}

IR              :   LINE+           
                ;

LINE            :   SEMICOLON (~NEWLINE)* NEWLINE
                ;

SEMICOLON       : ';'                   
                ;
NEWLINE         : '\n'              
                ;
WS              : (' ' | '\t')+ 
                  {$channel = HIDDEN;}
                ;

And here is my java class to run my lexer:

package com.ualberta.slmyers.cmput415.assign1;

import java.io.IOException;

import org.antlr.runtime.*;

public class Test {

public static void main(String[] args) throws RecognitionException,
        IOException {

    // create an instance of the lexer
    TestLex lexer = new TestLex(
            new ANTLRFileStream(
                    "/home/linux/workspace/Cmput415Assign1/src/com/ualberta/slmyers/cmput415/assign1/test3.s"));

    // wrap a token-stream around the lexer
    CommonTokenStream tokens = new CommonTokenStream(lexer);

    // when using ANTLR v3.3 or v3.4, un-comment the next line:
    tokens.fill();

    // traverse the tokens and print them to see if the correct tokens are
    // created
    int n = 1;
    for (Object o : tokens.getTokens()) {
        CommonToken token = (CommonToken) o;
        System.out.println("token(" + n + ") = "
                + token.getText().replace("\n", "\\n"));
        n++;
    }
}

}

credits to: http://bkiers.blogspot.ca/2011/03/2-introduction-to-antlr.html for the adapted code above.

This is my test file:

; token 1
; token 2
; token 3
; token 4

Note there is a newline character after the last '4'.

This is my output:

token(1) = ; token 1\n; token 2\n; token 3\n; token 4\n
token(2) = <EOF>

I'm expecting this as my output:

token(1) = ; token 1\n
token(2) = ; token 2\n
token(3) = ; token 3\n
token(4) = ; token 4\n
token(5) = <EOF>

Upvotes: 1

Views: 105

Answers (1)

slmyers
slmyers

Reputation: 767

OK I figured it out the problem was this line:

IR          :   LINE+           
            ;

which returned a one token comprised of many lines.

Upvotes: 1

Related Questions