CODEWITHSUNDEEP
pythoncpointersctypes
user1105224
user1105224

Reputation: 51

Python - passing pointers between C and Python with ctypes

I'm trying to write a small, modular program in Python that will dynamically load C functions and use them to execute computationally intensive code. In this program I am creating a couple of large matrices that I will be passing back and forth between my Python code to different C functions. I would prefer to pass these matrices by reference to avoid additional computational overhead.

I've tried reading through the Python docs for ctypes, but it doesn't seem to explain how to do this. I understand, for instance, that I can use byref() or pointer() to pass a pointer from Python to a C function, but how to I pass a pointer from an external C function back to Python? Given that variables are names in Python, is this just done "automatically" (for lack of a better term) when Python receives a value from a C function?

As a concrete example, this is what I'm trying to do (in pseudo-code):

foo = ctypes.CDLL("pathToFoo")
bar = ctypes.CDLL("pathToBar")

# Generate a large matrix by calling a C function.
reallyBigMatrix = foo.generateReallyBigMatrix()

# Pass reallyBigMatrix to another function and perform some operation
# on it. Since the matrix is really big, I would prefer to pass a 
# reference to this matrix to my next C function rather than passing
# the matrix by value.
modifiedReallyBigMatrix = bar.modifyReallyBigMatrix(reallBigMatrix)

Alternatively, I'm using Python and C in conjunction as I need an easy way to dynamically load C functions in my program. I may pass paths to different C files to my Python program so that the Python program will execute the same code on different functions. As an example, I may want to run my program two different ways: keep the same "generateReallyBigMatrix" function in both runs, but used a different "modifyReallyBigMatrix" program between run 1 and run 2. If there is an easy, cross-platform way to do this in C or C++ I would be happy to implement that solution rather than using ctypes and Python. However, I haven't been able to find a simple, cross-platform solution.

Upvotes: 3

Views: 2884

Answers (1)

John Zwinck
John Zwinck

Reputation: 249153

You've mentioned that you are writing all the code, both Python and C, from yourself. I suggest not using ctypes for this, as ctypes is best suited for using C libraries that cannot be modified.

Instead, write a module in C using the Python C API. It will expose a single function to start with, like this:

PyObject* generateReallyBigMatrix(void);

Now, instead of trying to return a raw C pointer, you can return any Python object that you like. Good choices here would be to return a NumPy array (using the NumPy C API), or to return a Python "buffer" (from which a NumPy array can be constructed in Python if desired).

Either way, once this function is written in C using the appropriate APIs, your Python code will be simple:

import foo
reallyBigMatrix = foo.generateReallyBigMatrix()

To do it using the NumPy C API, your C code will look like this:

PyObject* generateReallyBigMatrix(void)
{
    npy_intp dimension = 100;
    PyArray_Descr* descr;
    PyArray_DescrAlignConverter2("float64", &descr); // use any dtype

    PyObject* array = PyArray_Empty(1, &dimension, descr, 0/*fortran*/);
    Py_DECREF(descr);

    void* data = PyArray_DATA(array);
    // TODO: populate data

    return array;
}

static PyMethodDef methods[] = {
    {"generateReallyBigMatrix", generateReallyBigMatrix, METH_VARARGS, "doc"},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyMODINIT_FUNC initfoo(void)
{
    import_array(); // enable NumPy C API
    Py_InitModule("foo", methods);
}

Note that the NumPy C API requires a slightly strange initialization ritual. See also Numpy C API: Link several object files

You then compile the code as a shared library called foo.so (no lib prefix).

Upvotes: 2

Related Questions