CODEWITHSUNDEEP
pythonpicklerpy2
Marco Mene
Marco Mene

Reputation: 397

Memory leak when using pickle in python

I have a big pickle file containing hundreds of trained r-models in python: these are stats models built with the library rpy2.

I have a class that loads the pickle file every time one of its methods is called (this method is called several times in a loop). It happens that the memory required to load the pickle file content (around 100 MB) is never freed, even if there is no reference pointing to loaded content. I correctly open and close the input file. I have also tried to reload pickle module (and even rpy) at every iteration. Nothing changes. It seems that just the fact of loading the content permanently locks some memory.

Upvotes: 4

Views: 2918

Answers (2)

lgautier
lgautier

Reputation: 11545

I can reproduce the issue, and this is now an open issue in the rpy2 issue tracker: https://bitbucket.org/rpy2/rpy2/issues/321/memory-leak-when-unpickling-r-objects

edit: The issue is resolved and the fix is included in rpy2-2.7.5 (just released).

Upvotes: 4

mmghu
mmghu

Reputation: 621

If you follow this advice, please do so tentatively because I am not 100% sure of this solution but I wanted to try to help you if I could.

In Python the garbage collection doesn't use reference counting anymore, which is when Python detects how many objects are referencing an object, then removes it from memory when objects no longer are referencing it.

Instead, Python uses scheduled garbage collection. This means Python sets a time when it garbage collects instead of doing it immediately. Python switched to this system because calculating references can slow programs down (especially when it isn't needed)

In the case of your program, even though you no longer point to certain objects Python might not have come around to freeing it from memory yet, so you can do so manually using:

gc.enable() # enable manual garbage collection
gc.collect() # check for garbage collection

If you would like to read more, here is the link to Python garbage collection documentation. I hope this helps Marco!

Upvotes: 0

Related Questions