How to use meta-programming to generate code?

Question

Problem: In my system there are a lot of statistic calculators wrapped as classes. My job is to combine the values from these calculators into one single value called 'total'.

The 'model' that explains how these calculators should be combined is given as a table --- either a csv file or a table in MySQL

instance_name,accessor,argument,post_processing_function_name
fleet_statistics,getCash,2017,func1
city_statistics,getPopulation,2016,func2
....
....

My current solution: I wrote a python script to read the csv file and generate c++ code like this:

double computeTotal()
{
    double total = 0;
    total += func1(fleet_statistics->getCash(2017));
    total += func2(city_statistics->getPopulation(2016));
    ....
    ....
}

Using python to generate the code is very convenient, but the problem is that, it is no longer a 'pure' c++ program. I need to hack the makefile to make sure that the generated c++ file is up to date.

A more elegant solution is to use meta-programming to generate the code in c++ using c++ rather than python. Can anyone show me how to do this?

Jut to mention a few points:

1. the CSV file is generated for by some business-logics-related-scripts. It has a couple thousands of rows. It is impossible to write the c++ code manually.

2. The CSV file can change every time when we make a new release.

3. I use CSV format as an example. It could be a table in MySQL if that will make things easier

4. objects in the instance_name column are not necessarily from the same BaseClass. The accessor's function signature is variant as well. So it is not convenient to use a function pointer

Thanks!

Answer 1

but the problem is that, it is no longer 'pure' c++

So? How is that a real problem? Obviously your case is more complicated than simple "read csv and take total sum". This requires more complicated solutions. But dont make them more complicated than needed.

There is nothing wrong with the technique (i.e. C++ code generation via other script) you are using. While not common you can still see it elsewhere with the notable example of Google's protobuf.

So instead of looking for other complicated solutions (Jit compilation? Really? Even templates are often hard to follow) ask yourself: does my Python code generation work? Does it perform well enough? Is it easy to maintain? How much would the refactoring cost? These are real problems.

Also note that while makefiles are mostly used with C/C++ they are not strictly restricted to them. Actually any build process with one or many languages can be done with them. You can mix them. It's not hacking. It's simply a build process.

Answer 2

At first, you want to write some interpreter for the formulas in your CSV files. You probably could find a library doing exactly that. Otherwise, you need to implement such an interpreter (or embed some existing one, like Lua or Guile). Read absolutely some good book like the Dragon Book, SICP then Lisp In Small Pieces. You'll need to represent ASTs and environments (for variable bindings). If you have never been taught about compilers and interpreters you do need to spend several weeks reading about these (the techniques are mature but difficult, so you won't be able to reinvent all of them).

^{(I guess you have a good operating system, like Linux. Adapt my answer if not)}

If performance really matters because the formulas in your CSV files are complex and takes time to be computed, consider translating your ASTs (from the formulas) into some form of code using some JIT compilation library (e.g. libgccjit).

In some rare occasions (but probably not in your case) you might instead generate some C++ at runtime into some temporary file, then compile that as a plugin and dynamically load that temporary plugin (e.g. using dlopen & dlsym on Linux). Generally this is not worth the effort, because C++ compilers are really slow (at compile-time!). Quite often generating C code is more appropriate (but YMMV). But compiling to C or to C++ is never easy and takes months of development time.

I need to hack the makefile to make sure that the generated c++ file is up to date.

Not necessarily (needed to hack the Makefile, which is not a big deal). You could run g++ (or your C++ compiler, if it is not GCC) from your Python script or your C++ program. I don't know if (in your specific case) it is sensible to do that.

NB. Notice that -in the context of C++ specifically- meta-programming means (most often) not generating C++ code or some other code, but refers to template metaprogramming.

Answer 3

You mentioned in comments that the format of the input file is under your control. One way to use actual metaprogramming (more specifically preprocessor metaprogramming) would be to change the file's format into something suitable for using the X macro pattern. Something like this:

DATA_POINT(fleet_statistics,getCash,2017,func1)
DATA_POINT(city_statistics,getPopulation,2016,func2)

Then, in your C++ code, you'd do this:

double computeTotal()
{
    double total = 0;
    #define DATA_POINT(instance_name,accessor,argument,post_processing_function_name) \
      total += post_processing_function_name(instance_name->accessor(argument));
    #include "datafile"
    #undef DATA_POINT
}

This way, the C++ code being compiled is generated by the preprocessor from the data file.

However, you should definitely consider the opinions given in other comments and answers (which I share), that a parser could be more appropriate for your case. Normally, it's a good idea to keep code separated from data.

Answer 4

There should be no need to generate and compile C++ code when the CSV file changes.

All you should do is write C++ code once that parses the CSV file performs the calculations specified in the CSV file by invoking C++ functions on C++ objects.

So, roughly speaking, your fleet_statistics and city_statistics classes should implement some common interface that has a double getYearValueByName( String name, int year ) function, which checks the given name and invokes on itself a corresponding function of the same name to get a value for that year.

I understand that this approach is simplistic, and in reality things are more complicated than that; you will need to just keep extending that model, making it more complicated as necessary, until you have all the functionality needed to parse your CSV file and do as it requires.