Python Pandas Memory Accumulation

Question

When i try to store a Exception object on a column with apply function the memory start getting stacked up drastically and is not free up and system eventually hangs up.

Please find the working code snippet below to reproduce

import pandas as pd

def fx():
    try:
        raise Exception("ex")
    except Exception as e:
        return e

df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})

while True:
    df["test"] = df.apply(lambda row: fx(), axis=1)

But if the function fx is changed to return some actual value and not Exception it runs fine.

def fx(r):
    return "good"

Packages:

pandas==1.4.3

Answer 1

For analyzing programs that use too much memory, I tend to use Linux and to set some limits in the parent shell so that I get a sufficiently complete core and so that the process is not allowed to grow so large as to cause problems on the system:

$ ulimit -v 8388608
$ echo 0x37 >/proc/self/coredump_filter

Then I run the program in background and at some point before it crashes I gather a core:

$ python3 pandaswithexception.py &
[1] 444129
$ sleep 60 ; sudo gcore 444129; kill 444129
[sudo] password for tim: 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x000000000056b64b in _PyEval_EvalFrameDefault ()
warning: target file /proc/444129/cmdline contained unexpected null characters
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb720b04000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb720d38000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb720f4f000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb721165000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb72137d000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb7215d3000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb721b41000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb721d6a000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb721ff1000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb72226f000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb72255b000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb722764000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb722a97000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb72a71c000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb72ab25000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb72c9e9000.
warning: Memory read failed for corefile section, 1048576 bytes at 0x7fb72d074000.
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.444129
[Inferior 1 (process 444129) detached]

Then I open the core using chap and run a command to summarize the allocations by bytes and save the results in a file:

$ chap core.444129
chap> summarize used /sortby bytes /redirectSuffix usedByBytes
Wrote results to core.444129.usedByBytes
chap> 

The top few lines of that output file show that allocations matching the pattern %ContainerPython object (roughly, python objects that are subject to garbage collection) take lots of memory and this is broken down further by the size of those objects.

Pattern %ContainerPythonObject has 7421118 instances taking 0x4333a0d0(1,127,456,976) bytes.
   Matches of size 0x40 have 3227926 instances taking 0xc504580(206,587,264) bytes.
   Matches of size 0x1e0 have 375701 instances taking 0xabfb760(180,336,480) bytes.
   Matches of size 0x190 have 375672 instances taking 0x8f4eb80(150,268,800) bytes.
   Matches of size 0x238 have 187817 instances taking 0x65bcef8(106,680,056) bytes.
   Matches of size 0x218 have 187837 instances taking 0x60043b8(100,680,632) bytes.

The following command takes a sample of just the %ContainerPythonObject instances of size 0x1e0 and we can see from the output that they are instances of the python frame type:

chap> describe used %ContainerPythonObject /size 1e0 /geometricSample 20
Anchored allocation at 7fb6d6e06030 of size 1e0
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

Anchored allocation at 7fb6d6e115d0 of size 1e0
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

Anchored allocation at 7fb6d6f663f0 of size 1e0
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

Anchored allocation at 7fb6d868e3f0 of size 1e0
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

Anchored allocation at 7fb6f558d3f0 of size 1e0
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

5 allocations use 0x960 (2,400) bytes.

Starting from the last frame instance shown by the previous command, we can traverse backwards by incoming references to understand why that frame is kept in memory:

chap> describe allocation 7fb6f558d3f0 /skipUnfavoredReferences true /extend %ContainerPythonObject<- /extend %SimplePythonObject<- /extend %PyDictKeysObject<- /extend %PyDictValuesArray<- /extend ?@0<- /commentExtensions true /redirectSuffix OneFrame
Wrote results to core.444129.OneFrame

The first three groups of lines in that output show that the python frame object in the allocation at 0x7fb6f558d3f0 is referenced by a python traceback object which is referenced by a python Exception object. This makes sense because we know that we can get a stack trace from an exception.

Anchored allocation at 7fb6f558d3f0 of size 1e0
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame) 

# Allocation at 0x7fb6f558d3f0 is referenced by allocation at 0x7fb6f558b8f0.
Anchored allocation at 7fb6f558b8f0 of size 40 
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x9039a0 (traceback)

# Allocation at 0x7fb6f558b8f0 is referenced by allocation at 0x7fb6f55898a0.
Anchored allocation at 7fb6f55898a0 of size 50 
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 2 and python type 0x8fc4e0 (Exception)

Continuing down through that file we see a repeating sequence. The Exception is referenced by an allocation that chap doesn't recognize, which is referenced by a numpy.ndarray, which is referenced by a %PyDictKeysObject, which is referenced by a python dict object which is referenced by a %PyDictValuesArray, which is referenced by another python dict object which is referenced by a frame object which is referenced by another frame object which is referenced by another frame object which is referenced by another frame object, which is referenced by a traceback object which is referenced by another Exception. This sequence repeats many times but I will only show the first occurrence from the output file.

# Allocation at 0x7fb6f55898a0 is referenced by allocation at 0xb222d60.
Anchored allocation at b222d60 of size 38 

# Allocation at 0xb222d60 is referenced by allocation at 0x7fb6f5584390.
Anchored allocation at 7fb6f5584390 of size 60 
This allocation matches pattern SimplePythonObject.
This has reference count 1 and python type 0x7fb72d277e60 (numpy.ndarray)

# Allocation at 0x7fb6f5584390 is referenced by allocation at 0x7fb6f5586a80.
Anchored allocation at 7fb6f5586a80 of size b0 
This allocation matches pattern PyDictKeysObject.

# Allocation at 0x7fb6f5586a80 is referenced by allocation at 0x7fb6f558bbf0.
Anchored allocation at 7fb6f558bbf0 of size 40 
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x90ef00 (dict)

# Allocation at 0x7fb6f558bbf0 is referenced by allocation at 0x7fb6f5589940.
Anchored allocation at 7fb6f5589940 of size 50 
This allocation matches pattern PyDictValuesArray.
It contains values for a split python dict.

# Allocation at 0x7fb6f5589940 is referenced by allocation at 0x7fb6f558bc30.
Anchored allocation at 7fb6f558bc30 of size 40 
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x90ef00 (dict)

# Allocation at 0x7fb6f558bc30 is referenced by allocation at 0xb2223e0.
Anchored allocation at b2223e0 of size 238
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame) 

# Allocation at 0xb2223e0 is referenced by allocation at 0x7fb6f5587630.
Anchored allocation at 7fb6f5587630 of size 200
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 2 and python type 0x909420 (frame)

# Allocation at 0x7fb6f5587630 is referenced by allocation at 0x7fb6f558c800.
Anchored allocation at 7fb6f558c800 of size 190
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

# Allocation at 0x7fb6f558c800 is referenced by allocation at 0x7fb6f558d5d0.
Anchored allocation at 7fb6f558d5d0 of size 1e0
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

# Allocation at 0x7fb6f558d5d0 is referenced by allocation at 0x7fb6f558be30.
Anchored allocation at 7fb6f558be30 of size 40
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x9039a0 (traceback)

# Allocation at 0x7fb6f558be30 is referenced by allocation at 0x7fb6f5589a80.
Anchored allocation at 7fb6f5589a80 of size 50
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 2 and python type 0x8fc4e0 (Exception)

From that repeating pattern we understand the basic reason for the growth. Each time through the loop, even though we are populating the column with new Exception values, the Exception values from the current time through the loop each hold a traceback which indirectly holds at least one Exception value from the previous time through the loop which holds a traceback which holds at least one Exception from the loop iteration before that, and so on...

So the next question you might ask is, why does a frame indirectly reference an Exception? The answer is that frames are quite heavy weight objects and they hold both arguments and local variables. Exactly what the local variables are depends on what function is associated with the given frame.

So, for example, we can find out about the frame at 0xb2223e0, which references a dict that indirectly holds a dict that holds the numpy.darray that indirectly holds an Exception. By doing the following command we can see that the frame is associated with a call to apply_standard in lib/python3.8/site-packages/pandas/core/apply.py:

chap> describe allocation b2223e0 /extend ->%SimplePythonObject /skipUnfavoredReferences true /commentExtensions true
Anchored allocation at b2223e0 of size 238
This allocation matches pattern ContainerPythonObject.
This has a PyGC_Head at the start so the real PyObject is at offset 0x10.
This has reference count 1 and python type 0x909420 (frame)

# Allocation at 0xb2223e0 references allocation at 0x7fb71f374c90.
Anchored allocation at 7fb71f374c90 of size b0
This allocation matches pattern SimplePythonObject.
This has reference count 187814 and python type 0x9020c0 (code)

# Allocation at 0x7fb71f374c90 references allocation at 0x7fb71f3719b0.
Anchored allocation at 7fb71f3719b0 of size 40
This allocation matches pattern SimplePythonObject.
This has reference count 9 and python type 0x90e860 (str)
This has a string of length 14 containing
"apply_standard".

# Allocation at 0x7fb71f374c90 references allocation at 0x7fb71f371df0.
Anchored allocation at 7fb71f371df0 of size 40
This allocation matches pattern SimplePythonObject.
This has reference count 1 and python type 0x90e260 (bytes)

# Allocation at 0x7fb71f374c90 references allocation at 0x7fb71f372690.
Anchored allocation at 7fb71f372690 of size 30
This allocation matches pattern SimplePythonObject.
This has reference count 1 and python type 0x90e260 (bytes)

# Allocation at 0x7fb71f374c90 references allocation at 0x7fb71f3a9db0.
Anchored allocation at 7fb71f3a9db0 of size 80
This allocation matches pattern SimplePythonObject.
This has reference count 89 and python type 0x90e860 (str)
This has a string of length 65 containing
"/home/tim/.local/lib/python3.8/site-packages/pandas/core/apply.py".

6 allocations use 0x418 (1,048) bytes.

Using the source file path from the previous command, we can look at that file and see that apply_standard saves a copy of the old column in a local variable called "values":

def apply_standard(self) -> DataFrame | Series:
    f = self.f
    obj = self.obj

    with np.errstate(all="ignore"):
        if isinstance(f, np.ufunc):
            return f(obj)

        # row-wise access
        if is_extension_array_dtype(obj.dtype) and hasattr(obj._values, "map"):
            # GH#23179 some EAs do not have `map`
            mapped = obj._values.map(f)
        else:
            values = obj.astype(object)._values

If we look at the %PyDictKeysObject at 0x7fb6f5586a80, which references the numpy.ndarray at 0x7fb6f5584390, we can see that the key for that triple (which immediately precedes the value shown at offset 0x40) is a python str object "values".

chap> describe allocation 7fb6f5586a80 /showUpTo 100
Anchored allocation at 7fb6f5586a80 of size b0
This allocation matches pattern PyDictKeysObject.

 0:                1                8           5cf160                4
20:                1 ffffffffffffff00 98136e63b5fcf278     7fb72d533e70
40:     7fb6f5584390                0                0                0
60:                0                0                0                0
80:                0                0                0                0
a0:                0                0 
1 allocations use 0xb0 (176) bytes.
chap> describe 7fb72d533e70
Address 7fb72d533e70 is at offset 0 of
an anchored allocation at 7fb72d533e70 of size 40
This allocation matches pattern SimplePythonObject.
This has reference count 188823 and python type 0x90e860 (str)
This has a string of length 6 containing
"values".

chap> describe 7fb6f5584390
Address 7fb6f5584390 is at offset 0 of
an anchored allocation at 7fb6f5584390 of size 60
This allocation matches pattern SimplePythonObject.
This has reference count 1 and python type 0x7fb72d277e60 (numpy.ndarray)

I hope that the above is sufficiently clear about why the program grows each time through the loop, where ultimately instances of Exception are holding older instances of Exception.

With that observation, we can simplify the example even further, so it is not about pandas at all but only about Exception. I will leave it to the reader to run this:

def fx():
    try:
        raise Exception("ex")
    except Exception as e:
        return e

def gx(a):
    return fx()

v = None
while True:
    v = gx(v)