Python C API and data persistent in memory?

Question

I'm considering integrating some C code into a Python system (Django), and I was considering using the Python / C API. The alternative is two separate processes with IPC, but I'm looking into direct interaction first. I'm new to Python so I'm trying to get a feel for the right direction to take.

Is it possible for a call to a C initialiser function to malloc a block of memory (and put something in it) and return a handle to it back to the Python script (pointer to the start of the memory block). The allocated memory should remain on the heap after the init function returns. The Python script can then call subsequent C functions (passing as an argument the pointer to the start of memory) and the function can do some thinking and return a value to the Python script. Finally, there's another C function to deallocate the memory.

Assume that the application is single-threaded and that after the init function, the memory is only read from so concurrency isn't an issue. The amount of memory will be a few hundred megabytes.

• Is this even possible? Will Python let me malloc from the heap and allow it to stay there? Will it come from the Python process's memory? Will Python try and clear it up (i.e. does it do its own memory allocation and not expect any other processes to interfere with its address space)?
• Could I just return the byte array as a Python managed string (or similar datatype) and pass the reference back as an argument to the C call? Would Python be OK with such a large string?
• Would I be better off doing this with a separate process and IPC?

Answer 1

You can certainly use the C API to do what you want. You'll create a class in C, which can hold onto any memory it wants. That memory doesn't have to be exposed to Python at all if you don't want.

If you are comfortable building C DLLs, and don't need to perform Python operations in C, then ctypes might be your best bet.

Answer 2

Cython