CODEWITHSUNDEEP
pythoncnumpypython-c-api
Anger Density
Anger Density

Reputation: 332

NumPy C API extension causes excessive memory usage

I have written a C extension to NumPy to speed up some of my calculations, but I am getting increasing memory usage as I call the function repeatedly. I have trimmed down the function into a minimum example:

PyObject* memory_test_function(PyObject* self, PyObject* args)
{
    PyArrayObject *ang;
    int i;

    if (!PyArg_ParseTuple(args, "O", &ang)) return NULL;

    int L0 = (int) PyArray_DIMS(ang)[0];

    // ballooning memory usage 
    npy_intp final_out_dims[2] = {L0,1};
    PyObject *output_array;
    output_array = PyArray_SimpleNew(2, final_out_dims, NPY_FLOAT64);
    Py_INCREF(output_array);

    for (i=0;i<L0;i++) 
    {
        *(double *)PyArray_GETPTR2(output_array,i,0) = 
            tan(*(double *)PyArray_GETPTR2(ang,i,0));
    }

    return PyArray_Return(output_array);

    /* constant memory usage
    double sum=0.0;
    for (i=0;i<L0;i++) sum+=tan(*(double *)PyArray_GETPTR2(ang,i,0));
    return PyFloat_FromDouble(sum); */
}

The problem appears to be caused by creating the output array, as just returning a float without creating the array object is constant in memory. I suspect there is a problem with INCREF/DECREF, but I thought I was doing everything correctly. Calling this function repeatedly (a million or so times) makes memory usage increase linearly with time, which makes me think something is not being garbage collected correctly. Using gc manually doesn't help. Please let me know if there is anything obvious I'm missing!

Upvotes: 2

Views: 304

Answers (1)

Konrad Rudolph
Konrad Rudolph

Reputation: 546015

PyArray_SimpleNew calls (indirectly, probably) _Py_NewReference under the hood. This function sets the reference count on the newly created reference to 1.

Your subsequent Py_INCREF increases the ref count to 2 and thus makes sure that Python will never free this object, even if all references to it cease to exist, because its reference count will never drop to 0.

The Py_INCREF call is unnecessary because you’re not sharing the reference with any other object here, you’re just using it locally and then passing it to the caller.

Upvotes: 1

Related Questions