importing lexer grammar (ANTLR 4)

Question

I have a simple test grammar, which is split into two parts. The first part is the lexer:

lexer grammar TestLexer;

@lexer::header {
package de.test;
}

BOOL : ([Tt][Rr][Uu][Ee] | [Ff][Aa][Ll][Ss][Ee]) ;

AND : '&' ;
OR : '|' ;
IMPLIES : '=>' ;
IFF : '<=>' ;

NOT : '!' ;

WS : [ \t\n\r]+ -> skip ; // ignore whitespace

ANY : . ;

The second part is the actual grammar:

grammar TestGrammar;

@parser::header {
package de.test;
}

options {
tokenVocab=TestLexer;
}

formula  :
  BOOL
  | NOT formula
  | formula (AND | OR) formula
  | formula IMPLIES formula
  | formula IFF formula
;

As can be seen, the grammar includes the lexer using the option tokenVocab. I have this grammar in a maven repository with the following POM (A higher version than ANTLR 4.7.1 is not available in the nexus I can use. And I think that my problem is independent of the specific version of ANTLR 4.):

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>de.test</groupId>
  <artifactId>antlrTest</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>antlrTest</name>
  <url>http://maven.apache.org</url>
  <build>
    <plugins>
      <plugin>
        <groupId>org.antlr</groupId>
        <artifactId>antlr4-maven-plugin</artifactId>
        <version>4.7.1</version>
        <configuration>
          <sourceDirectory>${basedir}/grammar</sourceDirectory>
          <outputDirectory>${basedir}/src/main/java/de/test</outputDirectory>
          <visitor>true</visitor>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>antlr4</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  <dependencies>
    <dependency>
      <groupId>org.antlr</groupId>
      <artifactId>antlr4-runtime</artifactId>
      <version>4.7.1</version>
    </dependency>
  </dependencies>
  <properties>
    <java.version>1.8</java.version>
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>
  </properties>
</project>

When I now compile the grammar (using mvn clean install), the corresponding lexer (TestGrammarLexer.java) will be created without the corresponding package-information. Therefore, it cannot be used in my project. A manual change of the package-information is not an option as I need to update the grammar from time to time. And this forces a rebuild of the compiled classes, and therefore the package-information would be overwritten. I tried to include the lexer using ANTLR's import statement. But, this causes other problems, as I have a more complicated grammar where the corresponding lexer class will not even be generated at all.

When I instead merge the lexer with the grammar in one single g4-file, the compilation result is as desired:

grammar TestGrammar2;

@lexer::header {
package de.test;
}

@parser::header {
package de.test;
}

BOOL : ([Tt][Rr][Uu][Ee] | [Ff][Aa][Ll][Ss][Ee]) ;

AND : '&' ;
OR : '|' ;
IMPLIES : '=>' ;
IFF : '<=>' ;

NOT : '!' ;

WS : [ \t\n\r]+ -> skip ; // ignore whitespace

ANY : . ;

formula  :
  BOOL
  | NOT formula
  | formula (AND | OR) formula
  | formula IMPLIES formula
  | formula IFF formula
;

For now I am forced to merge the lexer with the parser in one single file. But, I would appreciate a separation since I want to reuse the lexer grammar in multiple different grammars. So, what am I doing wrong? Thank you very much for your help.

Answer 1

The version 4.9.2 is available in Maven central: https://mvnrepository.com/artifact/org.antlr/antlr4-maven-plugin/4.9.2 But, as you said, I don't think this is a version thing.

I recommend removing the @lexer::header and @parser::header definitions, and place your *.g4 files inside ./src/main/antlr4/de/test. The generated files would then get the package de.test.

In my POM files, I'm using the ANTLR plugin like this:

<plugin>
    <groupId>org.antlr</groupId>
    <artifactId>antlr4-maven-plugin</artifactId>
    <version>4.9.2</version>
    <configuration>
        <arguments>
            <argument>-visitor</argument>
        </arguments>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>antlr4</goal>
            </goals>
        </execution>
    </executions>
</plugin>