CODEWITHSUNDEEP
pythoncmemoryextension-methodsmemory-leaks
Néstor
Néstor

Reputation: 91

C-extension in Python - return Py_BuildValue() memory leak problem

I have a huge memory leak problem involving a C-extension I'm developing. In C, I have an array of doubles called A and an int variable called AnotherIntVariablethat I want to pass to Python. Well, in my C-extension module I do the following:

int i;  
PyObject *lst = PyList_New(len_A);  
PyObject *num;  
if(!lst)  
   return NULL;  
for(i=0;i<len_A;i++){  
   num=PyFloat_FromDouble(A[i]);  
   if(!num){  
      Py_DECREF(lst);  
      return NuLL;  
   }  
   PyList_SET_ITEM(lst,i,num);  
}  
free(A);  
return Py_BuildValue("Oi",lst,AnotherIntVariable)

So in Python i recieve this list and the int like this:

Pyt_A,Pyt_int=MyCModule.MyCFunction(...)

Where Pyt_A and Pyt_int are the list and the integer I get from my C-extension "MyCModule", from the function "MyCFunction" that I described earlier.

The problem is that, in Python, I use this Pyt_A array (so that's why I use Py_BuildValue instead of a simple return statement, to do an INCREF in order to save this variable for a moment from the garbage collector) but then I need to dereference it somehow in order to free that allocated memory. The problem is that I use the MyCFunction function several times, and this produces a memory leakage because I don't know how to dereference the array that I get in python in order to get rid of it.

I tried just returning the array by doing a return lst in the C part of the code instead of the Py_BuildValue("Oi",lst,AnotherIntVariable), but that only results in a Segmentation Fault when I try to use it in python (probably because the garbage collector did his work)...

...what am I missing here? Can anybody help me?

Upvotes: 9

Views: 5941

Answers (3)

ShadowRanger
ShadowRanger

Reputation: 155438

There's a simpler solution that what the other answers have been suggesting. Don't rearrange your code to decref after using the O format code (which takes a new strong reference to the passed object), just use the N format code, which steals your reference, avoiding unnecessary reference count manipulation entirely by transferring ownership. The only change needed for this is to change:

return Py_BuildValue("Oi",lst,AnotherIntVariable);

to:

return Py_BuildValue("Ni",lst,AnotherIntVariable);
//                    ^ only change

Py_BuildValue is pretty robust too; if it fails for any reason it continues parsing arguments to ensure reference counts are stolen and released appropriately. If the N argument (lst) is passed as NULL (usually because a function returning a new strong reference was called in the argument list and failed, setting an exception) it preserves that exception and returns NULL itself so the exception propagates as intended.

Upvotes: 2

ncoghlan
ncoghlan

Reputation: 41506

If you look at the documentation for Py_BuildValue (http://docs.python.org/3/c-api/arg.html#c.Py_BuildValue) you can see that under the O typecode, it says that the reference count of the passed in object is incremented by one (Note: an earlier section in that page describes the O typecode for PyArg_ParseTuple, which doesn't increment the reference count, but also isn't relevant here).

So, after the call to Py_BuildValue, the refcount for your list is 2, but you only want it to be 1.

Instead of returning the result of Py_BuildValue directly, save it to a PyObject pointer, decrement the lst reference count, then return your result.

You should be checking the result of the Py_BuildValue call anyway, since you also need to free num in the event that Py_BuildValue fails (i.e. returns NULL).

Upvotes: 13

N&#233;stor
N&#233;stor

Reputation: 31

Thanks for clearing it up Ignacio, now it makes so much sense! Finally, the solution was to, instead of returning directly the Py_BuildValue, do:

free(A);  
PyObject *MyResult = Py_BuildValue("Oi",lst,AnotherIntVariable);  
Py_DECREF(lst);  
return MyResult

It worked like a charm!

Upvotes: 3

Related Questions