CODEWITHSUNDEEP
c++
WhatsInAName
WhatsInAName

Reputation: 969

A good way to output array values from Python and then take them in through C++?

Due to annoying overflow problems with C++, I want to instead use Python to precompute some values. I have a function f(a,b) that will then spit out a value. I want to be able to output all the values I need based on ranges of a and b into a file, and then in C++ read that file and popular a vector or array or whatever's better.

  1. What is a good format to output f(a,b) in?
  2. What's the best way to read this back into C++?
  3. Vector or multidim array?

Upvotes: 0

Views: 190

Answers (3)

user9876
user9876

Reputation: 11102

If there is megabytes of data, then I would read the data in by memory mapping the data file, read-only. I would arrange things so I can use the data file directly, without having to read it all in at startup.

The reason for doing it this way is that you don't want to read megabytes of data at startup if you're only going to use some of the values. By using memory mapping, your OS will automatically read just the parts of the file that you need. And if you run low on RAM, your OS can reuse the memory allocated for that file without having to waste time writing it to the swap file.

If the output of your function is a single number, you probably just want an array of ints. You'll probably want a 2D array, e.g.:

#define DATA_SIZE (50 * 25)
typedef const int (*data_table_type)[50];
int fd = open("my_data_file.dat", O_RDONLY);
data_table_type data_table = (data_table_type)mmap(0, DATA_SIZE,
                                  PROT_READ, MAP_SHARED, fd, 0);
printf("f(5, 11) = %d\n", data_table[5][11]);

For more info on memory mapped files, see Wikipedia, or the UNIX mmap() function, or the Windows CreateFileMapping() function.

If you need more complicated data structures, you can put C/C++ structures and arrays into the file. But you can't embed pointers or any C++ class that has a virtual anything.

Once you've decided on how you want to read the data, the next question is how to generate it. struct.pack() is very useful for this - it will allow you to convert Python values into a properly-formatted Python string, which you can then write to a file.

Upvotes: 0

user9876
user9876

Reputation: 11102

You can use Python to write out C++ source code that contains your data. E.g:

def f(a, b):
    # Your function here, e.g:
    return pow(a, b, 65537)
num_a_values = 50
num_b_values = 50
# Write source file
with open('data.cpp', 'wt') as cpp_file:
    cpp_file.write('/* Automatically generated file, do not hand edit */\n\n')
    cpp_file.write('#include "data.hpp"\n')
    cpp_file.write('const int f_data[%d][%d] =\n'
                       % (num_a_values, num_b_values))
    cpp_file.write('{\n')
    for a in range(num_a_values):
        values = [f(a, b) for b in range(num_b_values)]
        cpp_file.write('  {' + ','.join(map(str, values)) + '},\n')
    cpp_file.write('}\n')
# Write corresponding header file
with open('data.hpp', 'wt') as hpp_file:
    hpp_file.write('/* Automatically generated file, do not hand edit */\n\n')
    hpp_file.write('#ifndef DATA_HPP_INCLUDED\n')
    hpp_file.write('#define DATA_HPP_INCLUDED\n')
    hpp_file.write('#define NUM_A_VALUES %d\n' % num_a_values)
    hpp_file.write('#define NUM_B_VALUES %d\n' % num_b_values)
    hpp_file.write('extern const int f_data[%d][%d];\n'
                              % (num_a_values, num_b_values))
    hpp_file.write('#endif\n')

You then compile the generated source code as part of your project. You can then use it by #including the header and accessing the f_data[] array directly.

This works really well for small to medium size data tables, e.g. icons. For larger data tables (millions of entries) some C compilers will fail, and you may find that the compile/link is unacceptably slow.

If your data is more complicated, you can use this same method to define structures.

[Based on Mark Ransom's answer, but with some style differences and more explanation].

Upvotes: 0

Mark Ransom
Mark Ransom

Reputation: 308206

You can use Python to write out a .h file that is compatible with C++ source syntax.

h_file.write('{')
for a in range(a_size):
    h_file.write('{' + ','.join(str(f(a, b)) for b in range(b_size)) + '},\n')
h_file.write('}')

You will probably want to modify that code to throw some extra newlines in, and in fact I have such code that I can show later (don't have access to it now).

Upvotes: 1

Related Questions