CODEWITHSUNDEEP
parsingantlr4lexer
intel_chris
intel_chris

Reputation: 826

Is there a way to insert phases between the lexer and parser in ANTLR

I am writing a lexer/parser for a language that allows abbreviations (and globs) for its keywords. And, I am trying to determine the best way to do it.

And one thought that occurs to me, is to insert a phase between the lexer and the parser, where the lexer recognizes the general class, e.g. is this a "command name" or is it an "option" and then passes those general tokens to a second phase which does further analysis and recognizes which command name it is and passes that on as the token type to the parser.

It will make the parser simple. I will only have to deal with well formed command names. Every token will be clear what it means.

It will keep the lexer simple. It will only have to divide things into classes. This is a simple name. This is a glob. This is an option name (starts with a dash).

The phase is the middle will also be relatively simple. The simple name (and option forms) will only have to deal with strings. The glob form can use standard glob techniques to match the glob against the legal candidates, which are in the tables for the simple names and options.

The question is how to insert that phase into ANTLR, so that I call the lexer and it creates tokens and the intermediate phase massages them and then the parser gets the tokens the intermediate phase has categorized.

Is there a known solution for this?

Something like:

lexer grammar simple
letter: [A-Z][a-z];
digit: [0-9];
glob-char: [*?];
name: letter (letter | digit)*;
option: '-'name;
glob: (glob-char|letter)(glob-char|letter|digit)*;
glob-option: '-'glob;

filter grammar name;
end: 'e' | 'end';
generate: 'ge' | 'generate';
goto: 'go' | 'goto';
help: 'h' | 'help';
if: 'i' | 'if';
then: 't' | 'then';

parser grammar simple;

The user (programmer writing the language I am parsing) need to be to write g*te and have if match generate.

The phase between the lexer and the parser when it sees a glob needs to look at the glob (and the list of keywords) and see if only one of them matches the glob and if so, return that keyword. The stuff I listed in the "filter grammar" is the stuff that builds the list of keywords globs can match. I have found code on the web that matches globs to a list of names. That part isn't hard.

And, I've since found in the ANTLR doc how to run arbitrary code on matching a token and how to change the resulting tokens type. (See my answer.)

Upvotes: 0

Views: 122

Answers (1)

intel_chris
intel_chris

Reputation: 826

It looks like you can use lexerCustomActions to achieve the desired effect. Something like the following.

in your lexer:

GLOB: [-A-Za-z0-9_.]* '*' [-A-Za-z0-9_.*]*  { setType(lexGlob(getText())); }

in your Java (or whatever language you are using code):

void int lexGlob(String origText()) {
   return xyzzy; // some code that computes the right kind of token type
}

Upvotes: 0

Related Questions