CODEWITHSUNDEEP
c++bisonflex-lexerwolfram-language
Benjamin Osenbach
Benjamin Osenbach

Reputation: 23

Implementing Wolfram language in flex, bison, c++

After seeing projects like mathics and symja, I am trying to implement an open-source parser for the Wolfram language using flex and bison in C++ . Invoking bison -d and flex++ don't raise any issues, but when I use g++, I get the following error message:

parser.tab.cpp:1242:16: error: use of undeclared identifier 'yylex'
  yychar = YYLEX;
           ^
parser.tab.cpp:598:16: note: expanded from macro 'YYLEX'
# define YYLEX yylex ()
           ^
1 error generated.

Here are my .lpp and .ypp files for reference

lexer.lpp

%{
#include <iostream>
#include "parser.tab.hpp"
using namespace std;

extern "C"
{
    int yylex(void);
}

%}

%option c++
%option noyywrap

%%
[1-9][0-9]*(.[0-9]*)?     { return NUM; }
"\["        { return LBRACE; }
"\]"        cout << "rBrace" << endl;
"\("        cout << "lParen" << endl;
"\)"        cout << "rParen" << endl;
"\{"        cout << "lBracket" << endl;
"\}"        cout << "rBracket" << endl;
","         cout << "comma" << endl;

"@@"        cout << "apply" << endl;
"Apply\["   cout << "apply" << endl;
"/@"        cout << "map" << endl;
"Map\["     cout << "map" << endl;
"/."        cout << "rule" << endl;

"==="       cout << "sameQ" << endl;
"SameQ\["   cout << "sameQ" << endl;

"+"         cout << "plus" << endl;
"-"         cout << "minus" << endl;
"*"         cout << "times" << endl;
"/"         cout << "divide" << endl;
"^"         cout << "power" << endl;
"Power\["   cout << "power" << endl;

--Abbreviated--

.           ECHO;
%%

int main()
{
    FlexLexer* lexer = new yyFlexLexer;
    while(lexer->yylex() != 0)
    ;

    return 0;
}

parser.ypp

%{
#include <iostream>
#include <string>

using namespace std;

extern "C"
{
    int yyparse(void);
}
void yyerror(const char *s);
%}

%union {
    double dval;
    char *str;
}

%token <dval> NUM;
%token <str> RBRACE;
%token <str> LBRACE;
%%

expr:
    NUM     { cout << $1 << endl;}
    | NUM "+" NUM { cout << $1 + $3}
    | NUM "-" NUM { cout << $1 - $3}
    | NUM "*" NUM { cout << $1 * $3}
    | NUM "/" NUM { cout << $1 / $3}
    ;
%%

int main(int argc, char **argv)
{
    yyparse();
}

void yyerror(const char *s)
{
    cout << s << endl;
}

Any help would be appreciated. Thank you!

Upvotes: 2

Views: 802

Answers (1)

rici
rici

Reputation: 241731

yylex is defined in the generated scanner and used (automatically) in the generated parser. Since the results are just ordinary C(++), there is no magic; if you use yylex in a file, you need to declare it in that file.

You might expect bison to include the declaration automatically, but it doesn't. For one thing, it would have no idea that you wanted to (unnecessarily and possibly futilely) wrap the declaration in extern "C" {...}.

Also, you're going to run into problems with the C++ interfaces. yylex is an member function in the flex C++ API, so you can't declare it as extern "C", and you also can't just call it as yylex in an external file.

YMMV, but personally I prefer to use the ordinary (stable and well-documented) C API, which will compile perfectly well as C++, avoiding the need for any extern "C" declarations.

If you want to avoid globals, use the reentrant scanner / pure parser interfaces.

Finally, flex comes with a perfectly good debug option which is available with almost zero cost by just specifying -d on the command line. A scanner generated with that flag will automatically output informative messages about every token scanned, and it is a lot easier to remove a command-line flag than to edit an entire flex description.

bison has a similar mechanism, but it isn't quite as automatic: you need to enable it when you generate the parser, and then you need to turn it on by setting a run-time flag. Both are well-documented in their respective manuals.

Upvotes: 1

Related Questions