Output AST from flex+bison to main.cpp

Question

Disclaimer:I'm a flex-bison noob, though I've done this tutorial :http://ds9a.nl/lex-yacc/cvs/lex-yacc-howto.html

Now, I'm building a video game for a project in OPENGL-C++. The main.cpp contains all the game graphics, logic etc(fairly manageable, so not a problem). Before the game starts, it needs to parse a config file(let's assume it's an arbitrary format, so INI and JSON APIs are out of question).

I know enough flex and bison to identify patterns in the files and create the AST(also assigning variables using the $ notation. Now, how do I make those variables available in main.cpp?

Answer 1

Depending on the complexity of your configuration language, you might be better off with a one-pass parser rather than creating an AST and then walking the tree. But both approaches are completely valid.

Probably you should spend a few minutes (or hours :) ) reading the bison manual. Here, I'll just focus on the general approach and the bison features you might use.

The most important one is the ability to pass extra parameters into your parser. In particular, you'll want to pass a reference or pointer to an object which will contain the parsed configuration. You need the additional output parameter because the parser itself will only return a success or failure indication (which you need as well).

So here's a simple example which just constructs a dictionary of names to strings. Note that unlike the author of the tutorial you mention, I prefer to compile both the scanner and the parser as C++, avoiding the need to extern "C" interfaces. This works fine with current versions of flex and bison, as long as you don't try to put non-POD objects onto the parser stack. Unfortunately, that means that we can't use std::string directly; we need to use a pointer (and we can't use a smart pointer either.)

file scanner.l

%{
  #include <string>
  #include "config.h"
  using std::string;
%}

%option noinput nounput noyywrap nodefault
%option yylineno
 // Set the output file to a C++ file. This could also be done on the
 // command-line
%option outfile="scanner.cc"

%%

"#".*                      ; /* Ignore comments */
[[:space:]]                ; /* Ignore whitespace */
[[:alpha:]̣_][[:alnum:]_]*  { yylval = new string(yytext, yyleng); return ID; }
[[:alnum:]_@]+             { yylval = new string(yytext, yyleng); return STRING; }
["][^"]*["]                { yylval = new string(yytext+1, yyleng-2); return STRING; }
.                          { return *yytext; }

Now the bison file, which only recognizes assignments. This requires bison v3; minor adjustments will be necessary to use it with bison v2.7.

config.y

%code requires {
  #include <map>
  #include <string>
  #include <cstdio>
  using Config = std::map<std::string, std::string>;

  // The semantic type is a pointer to a std::string
  #define YYSTYPE std::string*

  // Forward declarations
  extern FILE* yyin;
  extern int yylineno; 
  int yylex();
  // Since we've defined an additional parse parameter, it will also
  // be passed to yyerror. So we need to adjust the prototype accordingly.
  void yyerror(Config&, const char*);
}

 // Set the generated code filenames. As with the flex file, this is
 // probably
 // better done on the command line.
%output "config.cc"
%defines "config.h"

 // The parser takes an additional argument, which is a reference to the
 // dictionary
 // which will be returned.
%parse-param { Config& config }

%token ID STRING

 // If semantic values are popped off the stack as the result of error
 // recovery,
 // they will leak, so we need to clean up.
%destructor { delete $$; } ID STRING

%%

config: %empty
      | config assignment
      ;

assignment: ID '=' STRING { config[*$1] = *$3;
                            delete $1; delete $3;
                          }
          | ID '=' ID     { config[*$1] = config[*$3];
                            delete $1; delete $3;
                          } 

%%
// The driver would normally go into a separate file. I've put it here
// for simplicity.

#include <iostream>
#include <cstring>

void yyerror(Config& unused, const char* msg) {
  std::cerr << msg << " at line " << yylineno << '\n';
}

int main(int argc, const char** argv) {
  if (argc > 1) {
    yyin = fopen(argv[1], "r");
    if (!yyin) {
      std::cerr << "Unable to open " << argv[1] << ": "
                << strerror(errno) << '\n';
      return 1;
    }
  } else {
    yyin = stdin;
  }
  Config config;
  int rv = yyparse(config);
  if (rv == 0)
    for (const auto& kv : config)
      std::cout << kv.first << ": \"" << kv.second << "\"\n";
  return rv;
}

To compile:

flex scanner.l
bison config.y
g++ --std=c++11 -Wall config.cc scanner.cc

Try it out:

$ cat sample.config
a=17
b= @a_single_token@
c = "A quoted string"
d9 =
"Another quoted string"
$ ./config sample.config
a: "17"
b: "@a_single_token@"
c: "A quoted string"
d9: "Another quoted string"