CODEWITHSUNDEEP
python-3.xantlr4javadoc
Jayendra Parmar
Jayendra Parmar

Reputation: 774

parsing javadoc with antlr python

I am trying to parse the javadoc using python3 antlr4 runtime. The grammar has been obtained from here. The Parser and Lexer code has been generated with the help of this documentation. How can I generate the tree out of the code and parser,lexer ?

import antlr4


from grammars.javadoc.JavadocParser import JavadocParser
from grammars.javadoc.JavadocLexer import JavadocLexer
from grammars.javadoc.JavadocParserListener import JavadocParserListener


class MyJavaDocListener(JavadocParserListener):
  def enterDocumentation(self, ctx:JavadocParser.DocumentationContext):
    print(ctx.getRuleContext().start.line)
    print(ctx.getRuleContext().stop.line)
    print(ctx.getText())


if __name__ == '__main__':

  code = "class X{/**Some random comment*/public int testM(){return 42;}}"
  input_stream = antlr4.InputStream(code)
  lexer = JavadocLexer(input_stream)
  stream = antlr4.CommonTokenStream(lexer)
  parser = JavadocParser(stream)

  tree = parser.what?

  doclistener = MyJavaDocListener()
  walker = antlr4.ParseTreeWalker()
  walker.walk(doclistener, tree)

Upvotes: 1

Views: 666

Answers (1)

Bart Kiers
Bart Kiers

Reputation: 170178

The grammar you're using only works on Javadoc itself, not an entire Java source file containing Javadocs.

You use it like this:

code = "/**Some random comment*/"
input_stream = antlr4.InputStream(code)
lexer = JavadocLexer(input_stream)
stream = antlr4.CommonTokenStream(lexer)
parser = JavadocParser(stream)

# Call the `documentation` function. All parser rules are mapped to functions.
tree = parser.documentation()

doclistener = MyJavaDocListener()
walker = antlr4.ParseTreeWalker()
walker.walk(doclistener, tree)

If you want to extract them from Java source files, you would first need to parse them using a Java grammar/parser.

You could use the existing Java9 grammar and change the he last lexer rules:

COMMENT
    :   '/*' .*? '*/' -> channel(HIDDEN)
    ;

LINE_COMMENT
    :   '//' ~[\r\n]* -> channel(HIDDEN)
    ;

into this:

JAVADOC_COMMENT
    :   '/**' .*? '*/' -> channel(HIDDEN)
    ;

COMMENT
    :   '/*' .*? '*/' -> skip
    ;

LINE_COMMENT
    :   '//' ~[\r\n]* -> skip
    ;

Then create a custom listener that listens when an enterMethodDeclaration event occurs, and then get the previous token from the token-stream and see if that token is a hidden JAVADOC_COMMENT token.

A quick demo:

class JavaDocListener(Java9Listener):

    # methodDeclaration
    #   :   methodModifier* methodHeader methodBody
    #   ;
    def enterMethodDeclaration(self, ctx: Java9Parser.MethodDeclarationContext):
        previous_token_index = ctx.getSourceInterval()[0] - 1
        previous_token = ctx.parser.getTokenStream().tokens[previous_token_index]
        method_name = ctx.methodHeader().methodDeclarator().identifier().getText()
        javadoc = previous_token.text if previous_token.type == Java9Lexer.JAVADOC_COMMENT else None
        print('method: {}, javadoc: {}'.format(method_name, javadoc))


if __name__ == '__main__':

    code = """
        public class X {

          public static String mu() { return null; }

          /**
           * Some random comment
           */
          public int testM() {
            return 42;
          }
        }
    """
    input_stream = antlr4.InputStream(code)
    parser = Java9Parser(antlr4.CommonTokenStream(Java9Lexer(input_stream)))

    tree = parser.ordinaryCompilation()

    doc_listener = JavaDocListener()
    walker = antlr4.ParseTreeWalker()
    walker.walk(doc_listener, tree)

which will print:

method: mu, javadoc: None
method: testM, javadoc: /**
           * Some random comment
           */

Upvotes: 3

Related Questions