CODEWITHSUNDEEP
pythonmemory
Trampas
Trampas

Reputation: 431

identifying which variables are leaking memory in python using pympler

I have a python script that has a memory leak, I can see using pympler that my dictionary and list objects are growing by:

from pympler import tracker
tr = tracker.SummaryTracker()
tr.print_diff()

However I have not figured out how to list the variable names in my code that have the memory leak. Can someone help me list out the variable names and the code usage for that variable name?

Thanks

Upvotes: 5

Views: 2913

Answers (2)

Neil
Neil

Reputation: 1938

To list the variable names of the objects that have been created you can combine a call to Pympler's muppy.get_objects() function with my referrers library. For example:

import referrers
from pympler import summary, muppy

o1 = muppy.get_objects()
my_dict = {'a': [1]}
o2 = muppy.get_objects()

o1_ids = {id(obj) for obj in o1}
o2_ids = {id(obj): obj for obj in o2}
diff = [obj for obj_id, obj in o2_ids.items() if obj_id not in o1_ids]

summary.print_(summary.get_diff(summary.summarize(o1), summary.summarize(o2)), limit=2)

for obj in diff:
    print(
        referrers.get_referrer_graph(
            obj,
            exclude_object_ids=[id(o1), id(o2), id(diff), id(o2_ids)]
        )
    )

This will print a summary of the objects that have been created between o1 and o2 using Pympler, as well as the variable names that reference these objects using the referrers library. The output will look something like this:

  types |   # objects |   total size
======= | =========== | ============
   list |           2 |    782.27 KB
   dict |           1 |    232     B

╙── list instance (id=4452879296)
    ├─╼ o1 (global) (id=4452879296)
    ├─╼ obj (global) (id=4452879296)
    ├─╼ <module>.o1 (local) (id=4452879296)
    └─╼ <module>.obj (local) (id=4452879296)

╙── dict instance (id=4453670592)
    ├─╼ <module>.my_dict (local) (id=4453670592)
    ├─╼ obj (global) (id=4453670592)
    ├─╼ my_dict (global) (id=4453670592)
    └─╼ <module>.obj (local) (id=4453670592)

╙── list instance (id=4461235328)
    ├─╼ dict[a] (id=4453670592)
    │   ├─╼ <module>.my_dict (local) (id=4453670592)
    │   └─╼ my_dict (global) (id=4453670592)
    ├─╼ <module>.obj (local) (id=4461235328)
    └─╼ obj (global) (id=4461235328)

Upvotes: 0

DRz
DRz

Reputation: 1156

From Pympler's documentation:

Tracking the lifetime of objects of certain classes can be achieved with the Class Tracker. This gives insight into instantiation patterns and helps to understand how specific objects contribute to the memory footprint over time:

>>> from pympler import classtracker
>>> tr = classtracker.ClassTracker()
>>> tr.track_class(Document)
>>> tr.create_snapshot()
>>> create_documents()
>>> tr.create_snapshot()
>>> tr.stats.print_summary()
              active      1.42 MB      average   pct
   Document     1000    195.38 KB    200     B   13%

Is it what you were looking for?

Upvotes: 2

Related Questions