Antlr4 C# Application Tutorial/Example

Question

I want to use Antlr4 to parse some files in my C# application. I have been able to generate the parser and lexer files so far given my grammer. Now I would like to use read in the files and apply the parser and lexer to them. I have been searching for documentation on how to do that but I am coming up short. I have found some old examples using previous versions of Antlr but they don't appear to work for Antlr4. Any help would be appreciated. Thanks.

Answer 1

I am using Visual Studio 2019 Professional (latest version 16.7.3)

As for now ANTLR Language Support are not available for VS 2019. There is an unofficial version available https://github.com/tunnelvisionlabs/antlr4cs/issues/353 but it not complaint with VS2019 extension API (more info here: https://devblogs.microsoft.com/visualstudio/updates-to-synchronous-autoload-of-extensions-in-visual-studio-2019/)

You may try the following (steps for .net standard library)

1. Install VS extension AntlrVSIX 8.0 (using the Extension Manager)

2. create a .NET Standard Library project (MyLib.Parser.Grammar)

3. Created a dummy (.cs) class - not sure if its still necessary, there were some issues in the past if the project contained only grammar files

4. Reference the following packages (using Nuget)

   • Antlr4.Runtime.Standard
   • Antlr4BuildTasks

5. Add grammar files (.g4) e.g. you can used the grammar repository available here https://github.com/antlr/grammars-v4

6. Let's say you would like to parse TSQL (https://github.com/antlr/grammars-v4/tree/master/sql/tsql) - add TSqlParser.g4 and TSqllexer.g4 to your project

7. Edit the project file MyLib.Parser.Grammar.csproj, it should look something like

      <PropertyGroup>
        <TargetFramework>netstandard2.1</TargetFramework>
      </PropertyGroup>   
   
      <ItemGroup>
        <Antlr4 Include="TSqlLexer.g4">
          <Package>MyLib.Parser</Package>
          <Visitor>true</Visitor>
          <Error>false</Error>
          <Listener>true</Listener>      
        </Antlr4>
        <Antlr4 Include="TSqlParser.g4">
          <Package>MyLib.Parser</Package>
          <Visitor>true</Visitor>
          <Error>false</Error>
          <Listener>true</Listener>      
        </Antlr4>
      </ItemGroup>
   
      <ItemGroup>
        <PackageReference Include="Antlr4.Runtime.Standard" Version="4.8.0" />
        <PackageReference Include="Antlr4BuildTasks" Version="8.3.0" />
      </ItemGroup>
   
    </Project>
   

8. At this point when you build the MyLib.Parser.Grammar project the Antlr4BuildTasks tools will create the parser .cs files, but they will be available in the project bin folder (e.g. \MyLib.Parser.Grammar\bin\Debug\netstandard2.1)

9. Create another library project MyLib.Parser

10. Create project dependency so MyLib.Parser.Grammar is build before MyLib.Parser

11. Direct the output files from MyLib.Parser.Grammar to MyLib.Parser project by using the AntOutDir attribute and relative paths in the project definition. Now the Antlr4 section in the project file should like something like:

<ItemGroup>    
<Antlr4 Include="TSqlLexer.g4">
  <Package>MyLib.Parser</Package>
  <Visitor>true</Visitor>
  <Error>false</Error>
  <Listener>true</Listener>
  <AntOutDir>..\MyLib.Parser</AntOutDir>
</Antlr4>
<Antlr4 Include="TSqlParser.g4">
  <Package>MyLib.Parser</Package>
  <Visitor>true</Visitor>
  <Error>false</Error>
  <Listener>true</Listener>
  <AntOutDir>..\MyLib.Parser</AntOutDir>
</Antlr4>
</ItemGroup>

9. Now after rebuild, the generated .cs files should be added to the MyLib.Parser project automatically

Answer 2

here are a sample of ErrorListener

public class ErrorListener : BaseErrorListener
{
    public void SyntaxError(IRecognizer recognizer, int offendingSymbol, int line, int charPositionInLine, string msg, RecognitionException e)
    {
        Console.WriteLine("{0}: line {1}/column {2} {3}", e, line, charPositionInLine, msg);
    }
}

Answer 3

• In Visual Studio, go to Tools -> Extensions and Updates and search the Online section for "ANTLR Language Support" by Sam Harwell. More information can be found on the GitHub project site
  • This does a few things:
    • Adds Templates for the combined grammars.
    • Adds Syntax Highlighting
    • Adds an MSBuild target for the grammar to generate the parser.
• In your solution, set up your project structure like this:
  • Solution
    • GrammarProject
      • Calculator.g4
    • ImplementationProject
      • GeneratedFiles (All files in this folder are added as Links to files located in GrammarProject\obj\Debug)
        • CalculatorBaseListener.cs
        • CalculatorBaseVisitor.cs
        • CalculatorLexer.cs
        • CalculatorListener.cs
        • CalculatorParser.cs
        • CalculatorVistor.cs
      • MyCalcualtorImplementation.cs
• Write and Compile your grammar.
• In your folder for the Links to Generated Files, Right-Click the folder and click Add -> Existing Item
• Browse to Grammar Project\obj\Debug and select all the generated parser files.
• This next step is important. On the Add button there is a little drop-down arrow. Click the drop-down arrow and click "Add As Link".
  • This will add the generated files to the implementation project using a symbolic link instead of a direct copy.
  • This gives the added benefit of not having to remove and re-add the parser files if you have to change your grammar later.
• Once you have gotten this far, then you can inherit from GrammarProject.CalculatorBaseListener or GrammarProject.CalculatorBaseParser depending on what development pattern you have decided to use.

As a side note, "The Definitive ANTLR 4 Reference" by Terence Parr is an excellent resource to understand how ANTLR4 works and the difference development patterns. All the examples are in java, but the concepts apply to both Java and C#.

Answer 4

try with

using (StreamReader fileStream = new StreamReader(fileName)) {
    AntlrInputStream inputStream = new AntlrInputStream(fileStream);

    SearchLexer lexer = new SearchLexer(inputStream);
    CommonTokenStream commonTokenStream = new CommonTokenStream(lexer);
    SearchParser parser = new SearchParser(commonTokenStream);

    parser.RemoveErrorListeners();
    parser.AddErrorListener(new ErrorListener()); // add ours

    parser.root();
}