CODEWITHSUNDEEP
pythoncpython-extensions
KeyboardMan
KeyboardMan

Reputation: 1

C-extension in Python - pyObject called Py_DECREF ,reference is 0,but memory leak

this is my code.

PyObject *dataPyParams = PyList_New(0);
for (int i = 0; i < figdata.dataSetList.size(); i++)
{
    PyObject *pyParams = PyList_New(0);
    for (int j = 0; j < figdata.dataSetList[i].size(); j++)
    {
        //std::cerr << figdata.dataSetList[i][j] << "data\n";
        auto temp = Py_BuildValue("f", figdata.dataSetList[i][j]);
        PyList_Append(pyParams,temp);
        Py_DECREF(temp);    
    }
    PyList_Append(dataPyParams, pyParams);
    Py_DECREF(pyParams);
    
}
Py_DECREF(dataPyParams);

i called Py_DECREF(dataPyParams),dataPyParams reference is 0 ,but memory was not free. I try delete PyList_Append(pyParams,temp),this free up memory.It bothers me a lot.

Upvotes: 0

Views: 324

Answers (1)

ShadowRanger
ShadowRanger

Reputation: 155438

Python (and most languages) don't directly satisfy memory allocations from the OS, nor return allocations immediately on release. It allocates blocks of memory in bulk, and satisfies smaller requests by partitioning up the blocks. Even when all memory in a block is "freed", it's not always returned to the OS, but held in reserve against future allocations.

In this case, you're making float and list objects, both of which have "free lists" of their own, so freeing doesn't actually return them to the allocator, but to a simple stack of allocated, but unused, objects that the float and list constructors can pull from more cheaply than asking the allocator for more memory. Problem is, this also means the blocks those elements are on can't be returned to the OS at all, because parts of them are still allocated, at least from the allocator's point of view. You can clear those free lists by explicitly calling PyGC_Collect() (documented only by side-effect in a What's New doc), which might allow memory to be returned to the OS, but again, it's no guarantee. You'd probably also want to disable pymalloc (to avoid additional small object arenas that are even less likely to be handed back to the OS). Even with all that though, Python would be perfectly within its rights to hold onto much of the memory indefinitely to satisfy future allocation.

In short, this probably isn't a memory leak. You can use more advanced memory profiling tools (if nothing else, Python itself will tell you about arena usage if you define PYTHONMALLOCSTATS=1 in your environment prior to launch), but the Task Manager can only see what the OS sees, not the internal memory management that Python itself layers on top of the raw, bulk OS memory allocation.

A simple test to see if this leaks with no external tooling would be to run this exact code (including the cleanup) multiple times. If the memory keeps going up by a large fixed amount each time it's run, then yeah, it's probably a leak. But more likely, you'll find that the first run consumes a significant amount of memory, but subsequent runs add little or no memory usage (some might get used due to allocation ordering and allocation alignment issues, but it would be pretty small), as they're drawing on the memory freed (in userland, not to the OS) rather than asking the OS for more memory.

Upvotes: 2

Related Questions