CODEWITHSUNDEEP
pythonparsinggrammarlark-parser
Branquif
Branquif

Reputation: 53

lark-parser indented DSL and multiline documentation strings

I'm trying to implement a record definition DSL using lark. It is based on indentation, which makes things a bit more complex.

Lark is a great tool, but I'm facing some dificulteis.

Here is a snippet of the DSL I'm implementing:

record Order :
    """Order record documentation
    should have arbitrary size"""

    field1 Int
    field2 Datetime:
        """Attributes should also have
        multiline documentation"""

    field3 String "inline documentation also works"

and here is the grammar used:

?start: (_NEWLINE | redorddef)*

simple_type: NAME

multiline_doc:  MULTILINE_STRING _NEWLINE
inline_doc: INLINE_STRING

?element_doc:  ":" _NEWLINE _INDENT multiline_doc _DEDENT | inline_doc

attribute_name: NAME
attribute_simple_type: attribute_name simple_type [element_doc] _NEWLINE
attributes: attribute_simple_type+
_recordbody: _NEWLINE _INDENT [multiline_doc] attributes _DEDENT
redorddef: "record" NAME ":" _recordbody



MULTILINE_STRING: /"""([^"\\]*(\\.[^"\\]*)*)"""/
INLINE_STRING: /"([^"\\]*(\\.[^"\\]*)*)"/

_WS_INLINE: (" "|/\t/)+
COMMENT: /#[^\n]*/
_NEWLINE: ( /\r?\n[\t ]*/ | COMMENT )+

%import common.CNAME -> NAME
%import common.INT

%ignore /[\t \f]+/  // WS
%ignore /\\[\t \f]*\r?\n/   // LINE_CONT
%ignore COMMENT
%declare _INDENT _DEDENT

It works fine for multiline string docs for the record definition, works fine for inline attribute definition, but doesn't work for attribute multiline string doc.

The code I use to execute is this:

import sys
import pprint

from pathlib import Path

from lark import Lark, UnexpectedInput
from lark.indenter import Indenter

scheman_data_works = '''
record Order :
        """Order record documentation
        should have arbitrary size"""

        field1 Int
        # field2 Datetime:
        #   """Attributes should also have
        #   multiline documentation"""

        field3 String "inline documentation also works"
'''

scheman_data_wrong = '''
record Order :
        """Order record documentation
        should have arbitrary size"""

        field1 Int
        field2 Datetime:
                """Attributes should also have
                multiline documentation"""

        field3 String "inline documentation also works"
'''
grammar = r'''

?start: (_NEWLINE | redorddef)*

simple_type: NAME

multiline_doc:  MULTILINE_STRING _NEWLINE
inline_doc: INLINE_STRING

?element_doc:  ":" _NEWLINE _INDENT multiline_doc _DEDENT | inline_doc

attribute_name: NAME
attribute_simple_type: attribute_name simple_type [element_doc] _NEWLINE
attributes: attribute_simple_type+
_recordbody: _NEWLINE _INDENT [multiline_doc] attributes _DEDENT
redorddef: "record" NAME ":" _recordbody



MULTILINE_STRING: /"""([^"\\]*(\\.[^"\\]*)*)"""/
INLINE_STRING: /"([^"\\]*(\\.[^"\\]*)*)"/

_WS_INLINE: (" "|/\t/)+
COMMENT: /#[^\n]*/
_NEWLINE: ( /\r?\n[\t ]*/ | COMMENT )+

%import common.CNAME -> NAME
%import common.INT

%ignore /[\t \f]+/  // WS
%ignore /\\[\t \f]*\r?\n/   // LINE_CONT
%ignore COMMENT
%declare _INDENT _DEDENT

'''

class SchemanIndenter(Indenter):
    NL_type = '_NEWLINE'
    OPEN_PAREN_types = ['LPAR', 'LSQB', 'LBRACE']
    CLOSE_PAREN_types = ['RPAR', 'RSQB', 'RBRACE']
    INDENT_type = '_INDENT'
    DEDENT_type = '_DEDENT'
    tab_len = 4

scheman_parser = Lark(grammar, parser='lalr', postlex=SchemanIndenter())
print(scheman_parser.parse(scheman_data_works).pretty())
print("\n\n")
print(scheman_parser.parse(scheman_data_wrong).pretty())

and the result is:

redorddef
Order
multiline_doc """Order record documentation
        should have arbitrary size"""
attributes
    attribute_simple_type
    attribute_name    field1
    simple_type       Int
    attribute_simple_type
    attribute_name    field3
    simple_type       String
    inline_doc        "inline documentation also works"




Traceback (most recent call last):
File "schema_parser.py", line 83, in <module>
    print(scheman_parser.parse(scheman_data_wrong).pretty())
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/lark.py", line 228, in parse
    return self.parser.parse(text)
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/parser_frontends.py", line 38, in parse
    return self.parser.parse(token_stream, *[sps] if sps is not NotImplemented else [])
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/parsers/lalr_parser.py", line 68, in parse
    for token in stream:
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/indenter.py", line 31, in process
    for token in stream:
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/lexer.py", line 319, in lex
    for x in l.lex(stream, self.root_lexer.newline_types, self.root_lexer.ignore_types):
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/lexer.py", line 167, in lex
    raise UnexpectedCharacters(stream, line_ctr.char_pos, line_ctr.line, line_ctr.column, state=self.state)
lark.exceptions.UnexpectedCharacters: No terminal defined for 'f' at line 11 col 2

        field3 String "inline documentation also
^

I undestand indented grammars are more complex, and lark seems to make it easier, but cannot find the mistake here.

PS: I also tried pyparsing, without success wit this same scenario, and would be hard for me to move to PLY, given the amount of code that will probably be needed.

Upvotes: 5

Views: 1736

Answers (2)

PaulMcG
PaulMcG

Reputation: 63782

You mentioned trying pyparsing, otherwise I would have left your question alone.

Whitespace-sensitive parsing is not great with pyparsing, but it does make an effort on this kind of case, using pyparsing.indentedBlock. There is some level of distress in writing this, but it can be done.

import pyparsing as pp

COLON = pp.Suppress(':')
tpl_quoted_string = pp.QuotedString('"""', multiline=True) | pp.QuotedString("'''", multiline=True)
quoted_string = pp.ungroup(tpl_quoted_string | pp.quotedString().addParseAction(pp.removeQuotes))
RECORD = pp.Keyword("record")
ident = pp.pyparsing_common.identifier()

field_expr = (ident("name")
              + ident("type") + pp.Optional(COLON)
              + pp.Optional(quoted_string)("docstring"))

indent_stack = []
STACK_RESET = pp.Empty()
def reset_indent_stack(s, l, t):
    indent_stack[:] = [pp.col(l, s)]
STACK_RESET.addParseAction(reset_indent_stack)

record_expr = pp.Group(STACK_RESET
                       + RECORD - ident("name") + COLON + pp.Optional(quoted_string)("docstring")
                       + (pp.indentedBlock(field_expr, indent_stack))("fields"))

record_expr.ignore(pp.pythonStyleComment)

If your example is written to a variable 'sample', do:

print(record_expr.parseString(sample).dump())

And get:

[['record', 'Order', 'Order record documentation\n    should have arbitrary size', [['field1', 'Int'], ['field2', 'Datetime', 'Attributes should also have\n        multiline documentation'], ['field3', 'String', 'inline documentation also works']]]]
[0]:
  ['record', 'Order', 'Order record documentation\n    should have arbitrary size', [['field1', 'Int'], ['field2', 'Datetime', 'Attributes should also have\n        multiline documentation'], ['field3', 'String', 'inline documentation also works']]]
  - docstring: 'Order record documentation\n    should have arbitrary size'
  - fields: [['field1', 'Int'], ['field2', 'Datetime', 'Attributes should also have\n        multiline documentation'], ['field3', 'String', 'inline documentation also works']]
    [0]:
      ['field1', 'Int']
      - name: 'field1'
      - type: 'Int'
    [1]:
      ['field2', 'Datetime', 'Attributes should also have\n        multiline documentation']
      - docstring: 'Attributes should also have\n        multiline documentation'
      - name: 'field2'
      - type: 'Datetime'
    [2]:
      ['field3', 'String', 'inline documentation also works']
      - docstring: 'inline documentation also works'
      - name: 'field3'
      - type: 'String'
  - name: 'Order'

Upvotes: 0

Erez
Erez

Reputation: 1430

The bug comes from misplaced _NEWLINE terminals. Generally, it's recommended to make sure rules are balanced, in terms of their role in the grammar. So here's how you should have defined element_doc:

?element_doc:  ":" _NEWLINE _INDENT multiline_doc _DEDENT
            | inline_doc _NEWLINE

Notice the added newline, which means that no matter which of the two options the parser takes, it ends in a similar state, syntax-wise (_DEDENT also matches a newline).

The second change, as a consequence of the first one, is:

attribute_simple_type: attribute_name simple_type (element_doc|_NEWLINE)

As element_doc already handles newlines, we shouldn't try to match it twice.

Upvotes: 4

Related Questions