In ANTLR 3, how do I generate a lexer (and parser) at runtime instead of ahead of time?

Question

I want to generate an antlr lexer at runtime -- that is, generate the grammar and from the grammar generate the lexer class, and its supporting bits at runtime. I am happy to feed it into the the java compiler, which is accessible at runtime.

Answer 1

ANTRL 4

Since I couldn’t find a specific topic for ANTLR 4, I decided to share a solution similar to Bart Kiers' answer but tailored for ANTLR 4. This solution is useful if you need to dynamically generate an ANTLR lexer and parser at runtime. However, be cautious: this approach leverages Java Reflection and writes temporary files to the file system, which can be considered "dirty" and risky in certain environments.

public final class DynamicLexerAndParser {

    /**
     * Temporary directory.
     * Where the generated ANTLR lexer and parser will be stored.
     */
    private final Path temp;

    /**
     * ANTLR grammar text.
     */
    private final String grammar;

    /**
     * Top rule name. Required to build a parse tree.
     */
    private final String top;

    /**
     * Constructor.
     * @param temp Temporary directory where the generated ANTLR lexer and parser will be stored.
     * @param grammar ANTLR grammar text.
     * @param top Top rule name.
     */
    DynamicLexerAndParser(final Path temp, final String grammar, final String top) {
        this.temp = temp;
        this.grammar = grammar;
        this.top = top;
    }

    /**
     * Dynamically generate lexer and parser.
     * @param code Code example for provided grammar.
     */
    public void generate(final String code) throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, IOException {
            final String name = this.grammarName();
            // Save grammar to a temp dir
            final Path gpath = this.temp.resolve(String.format("%s.g4", name));
            Files.write(gpath, this.grammar.getBytes(StandardCharsets.UTF_8));
            // Generate Parser.java and Lexer.java
            final Tool tool = new Tool(new String[]{gpath.toString()});
            tool.processGrammarsOnCommandLine();
            // Compile Parser.java and Lexer.java
            ToolProvider.getSystemJavaCompiler().run(
                System.in,
                System.out,
                System.err,
                Files.list(this.temp).filter(Files::isRegularFile)
                    .filter(java -> java.getFileName().toString().endsWith(".java"))
                    .map(Path::toString)
                    .toArray(String[]::new)
            );
            // Generated lexer
            final Lexer lexer = this.lexer(this.load(String.format("%sLexer", name)), 
            // Generated parser
            final Parser parser = this.parser(this.load(String.format("%sParser", name)), lexer);
    }

    /**
     * Create parser instance.
     * @param cparser Loaded parser class.
     * @param lexer Lexer instance.
     * @return Parser instance.
     */
    private Parser parser(final Class<?> cparser, final Lexer lexer) throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException {
            final Constructor<?> constructor = cparser.getDeclaredConstructor(TokenStream.class);
            return (Parser) constructor.newInstance(new CommonTokenStream(lexer));
    }

    /**
     * Create lexer instance.
     * @param clexer Loaded lexer class.
     * @param code Code to parse.
     * @return Lexer instance.
     */
    private Lexer lexer(final Class<?> clexer, final String code) throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException {
            final Constructor<?> constructor = clexer.getDeclaredConstructor(CharStream.class);
            return (Lexer) constructor.newInstance(CharStreams.fromString(code));
    }

    /**
     * Get grammar name.
     * @return Grammar name.
     */
    String grammarName() {
        final Matcher matcher = Pattern.compile("grammar\\s+(\\w+);").matcher(this.grammar);
        if (matcher.find()) {
            return matcher.group(1);
        } else {
            throw new IllegalStateException("Grammar name not found");
        }
    }

    /**
     * Load class from the temp directory.
     * @param name Class name.
     * @return Loaded class.
     */
    private Class<?> load(String name) {
        try {
            return new URLClassLoader(
                new URL[]{this.temp.toFile().toURI().toURL()}
            ).loadClass(name);
        } catch (final MalformedURLException | ClassNotFoundException exception) {
            throw new IllegalStateException(
                "Something went wrong during class loading",
                exception
            );
        }
    }
}

Answer 2

Here's a quick and dirty way to:

1. generate a combined (!) ANTLR grammar .g file given a String as grammar-source,
2. and create a Parser & Lexer from this .g file,
3. compile the these Parser & Lexer .java files,
4. create instances of the Parser & Lexer classes and invoke the entry point of the parser.

Main.java

import java.io.*;
import javax.tools.*;
import java.lang.reflect.*;
import org.antlr.runtime.*;
import org.antlr.Tool;

public class Main {

    public static void main(String[] args) throws Exception {

        // The grammar which echos the parsed characters to theconsole,
        // skipping any white space chars.
        final String grammar =
                "grammar T;                                                  \n" +
                "                                                            \n" +
                "parse                                                       \n" +
                "  :  (ANY {System.out.println(\"ANY=\" + $ANY.text);})* EOF \n" +
                "  ;                                                         \n" +
                "                                                            \n" +
                "SPACE                                                       \n" +
                "  :  (' ' | '\\t' | '\\r' | '\\n') {skip();}                \n" +
                "  ;                                                         \n" +
                "                                                            \n" +
                "ANY                                                         \n" +
                "  :  .                                                      \n" +
                "  ;                                                           ";
        final String grammarName = "T";
        final String entryPoint = "parse";

        // 1 - Write the `.g` grammar file to disk.
        Writer out = new BufferedWriter(new FileWriter(new File(grammarName + ".g")));
        out.write(grammar);
        out.close();

        // 2 - Generate the lexer and parser.
        Tool tool = new Tool(new String[]{grammarName + ".g"});
        tool.process();

        // 3 - Compile the lexer and parser.
        JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
        compiler.run(null, System.out, System.err, "-sourcepath", "", grammarName + "Lexer.java");
        compiler.run(null, System.out, System.err, "-sourcepath", "", grammarName + "Parser.java");

        // 4 - Parse the command line parameter using the dynamically created lexer and 
        //     parser with a bit of reflection Voodoo :)
        Lexer lexer = (Lexer)Class.forName(grammarName + "Lexer").newInstance();
        lexer.setCharStream(new ANTLRStringStream(args[0]));
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        Class<?> parserClass = Class.forName(grammarName + "Parser");
        Constructor parserCTor = parserClass.getConstructor(TokenStream.class);
        Parser parser = (Parser)parserCTor.newInstance(tokens);
        Method entryPointMethod = parserClass.getMethod(entryPoint);
        entryPointMethod.invoke(parser);
    }
}

Which, after compiling and running it like this (on *nix):

java -cp .:antlr-3.2.jar Main "a b    c"

or on Windows

java -cp .;antlr-3.2.jar Main "a b    c"

, produces the following output:

ANY=a
ANY=b
ANY=c

Answer 3

You'll have to use org.antlr.Tool() class to get it working.

You can check ANTLRWorks source code on github to have an idea how to use it, specifically the generate() method here:

ErrorListener el = ErrorListener.getThreadInstance();
ErrorManager.setErrorListener(el);

String[] params;
if(debug)
    params = new String[] { "-debug", "-o", getOutputPath(), "-lib", window.getFileFolder(), window.getFilePath() };
else
    params = new String[] { "-o", getOutputPath(), "-lib", window.getFileFolder(), window.getFilePath() };

new File(getOutputPath()).mkdirs();

Tool antlr = new Tool(Utils.concat(params, AWPrefs.getANTLR3Options()));
antlr.process();

boolean success = !el.hasErrors();
if(success) {
    dateOfModificationOnDisk = window.getDocument().getDateOfModificationOnDisk();
}
lastError = el.getFirstErrorMessage();
el.clear();
ErrorManager.removeErrorListener();
return success;

Answer 4

Have you tried calling org.antlr.Tool.main(String[]) with an appropriate String[] argument?

If that's too cumbersome, you could reverse engineer the Tool class (source code) to figure out how it works, and how to do the specific tasks you need to do.