CODEWITHSUNDEEP
pythonmemory-leakspython-napari
Pepv
Pepv

Reputation: 171

Python process not cleaning memory as expected (memory-leak)

Summary:

Python process is not managing memory as expected, resulting in the process getting killed.

Details:

I'm making an app in python that manages huge image data (hundreds of 32bit 3000x3000 px images). I'm trying to manage the data in the most storage-efficient and memory-efficient way by following the OOP principles, saving the data in optimized formats, loading the data in minimal batches and keeping almost all variables out of the "main" scope.

However, I'm facing a problem that I'm unable to diagnose. After running a method, the memory usage skyrockets from 40% to 80%. This method opens multiple stacks of images in napari, so it is expected to use that much memory (nevertheless, I should optimize it).

The issue arises when exiting this method, as the memory is not freed. This means that running the method twice or performing any other intense work afterwards fills up the memory and makes the program crash. The method runs out of the "main" scope. I've printed the local and global variables from the "main" scope before and after running this method:

Before the issue:

before

After the issue:

after

I already tried:

Running gc.collect from the main scope and making sure from the debugger that no napari-related object exists after the execution of the method.

Maybe there is some variable not show by locals().items() or globals().items(), or maybe I simply don't understand how Python allocates memory at all. This is my first time dealing with memory management and garbage collection in Python, so any information will be highly appreciated.

Edit:

I've been playing with objgraph to track the memory leak, and I found that the Garbage Collector is not removing napari-related objects upong closing napari. This means that I should move this question to napari's Issues page, on github. However, it would be highly appreciated if someone knew of a way of cleaning all module-related objects, so I could just dump all the napari leftover trash. The alternative for the moment is just closing and running the script, however, this is far from desired.

Upvotes: 0

Views: 1041

Answers (0)

Related Questions