CODEWITHSUNDEEP
xtextlexer
Amine Ben
Amine Ben

Reputation: 65

Per-rule hidden terminals are not captured in the internal.tokens file generated by Xtext

I have the following minimalistic grammar:

grammar org.example.minimalDSL hidden (WS, SL_COMMENT, ML_COMMENT)
...
Class: 
    (Documentation=documentation)?
    'class' name=ValidId '{' (attributes+=Attribute ';') '}';
Attribute: 
     type=[Type | ID] (multiplicity=Multiplicity)? name=PropertyName;
Documentation hidden(WS, HIDDEN_ASTERISK):
     '/**' {Documentation}((tags+=Tag)*) '*/';
Tag: 
'@' tagName=ID (tagValue= STRING)? 
Multiplicity hidden():
    '[' (bounds+=UnlimitedNatural ('..' bounds+=UnlimitedNatural)?)? ']');

terminal HIDDEN_ASTERISK: ('*') !('*/');
UnlimitedNatural returns EInt hidden():
    INT | '*';

I am trying to add Javadoc style to an existing DSL. A documentation starts with '/**' and ends "/". The documentation block should ignore whitespaces and '' when it is not part of the starting and ending tokens.

The HIDDEN_ASTERICKS terminal results in Xtext not generating '*' in the [DSL]Internal.tokens file. This results in producing erroneous tokens when trying to parse a text containing '*' bounds, e.g. [*].

While it makes total sense not to capture hidden terminal in the tokens files, I was naively expecting the per-rule hidden tokens to be captured in [DSL]internal.tokens. These local tokens can then be ignored only at the rule application time.

I am missing something? I am using Xtext v2.28.0. Is this a known issue that is solved in newer versions?

Thank you in advance for your answers or explanations. Kind regards,

Upvotes: 0

Views: 51

Answers (0)

Related Questions