CODEWITHSUNDEEP
python-3.xmatplotlibmemory-leaksipythonspyder
Vahid S. Bokharaie
Vahid S. Bokharaie

Reputation: 997

Spyder does not release memory for matplotlib plots

I use Spyder 3.3.6, with Pyhton 3.7.3 over IPyhton 7.7.0 cosole.

For different reasons, Spyder has always been my IDE of choice, but now I have decided to drop it altogether and stick to command prompt only.

The reason is that I have seen a serious memory leak while using matplotlib which is caused by Spyder.

Memory increases linearly until all my 128 GB of memory is consumed to the extend that my Ubuntu machine halted completely and I had to hard reset it. I have had this issue in the past with matplotlib and the solution I found, things like using

plt.close('all') or plt.close(fig) or gc.collect()

helped to some extent, but this time none worked (I should add that I am using MNE library to plot a long list of EEG raw files, hence I cannot try every possible solution that exists for such matplotlib memory issues).

But when I ran the same code using both python and ipython in command prompt, a simple

plt.close(fig)

solved the issue and the memory stays in the order of a few GB during the whole run. Hence, the only logical conclusion is that Spyder somehow messes with the memory management.

So, I wrote this question in case somebody knows a solution to this issue or if not, other people who use Spyder and probably have wasted hours trying to find a python solution would know the issue is in Spyder, not python.

Upvotes: 3

Views: 1701

Answers (1)

Markus Dutschke
Markus Dutschke

Reputation: 10606

For me, using gc.collect resolved the issue. I am using: Spyder 3.3.6, Python 3.7.6, Ipython 7.11.1.

Minimal working/crashing example

The code below does not crash but can be modified to cause a memory leak by commenting/uncommenting the two lines indicated.

import numpy as np
import matplotlib.pylab as plt
import os
import psutil
import gc
process = psutil.Process(os.getpid())
def str_mem_usage():
    mu = process.memory_info().rss / 1024**2
    return 'Memory usage: {:.2f} MB'.format(mu)
    
arrs = []
for ii in range(10):
    print('it', ii, '\t', str_mem_usage())
    n = 10000
    arr = np.random.rand(n**2).reshape(n, n)
    #arrs += [arr]  # activate to cause memory leak, obviously
    
    plt.ioff()
    fig, ax = plt.subplots(1, 1)
    ax.imshow(arr)
    #plt.savefig('tmp.pdf')  # irrelevant for memory leak
    plt.close(fig)
    plt.ion()
    
    gc.collect()  # deactivate to cause memory leak

This behavior is the same for execution in Spyder or calling Python or iPython from the Anaconda distribution.

Output (no memory leak)

it 0     Memory usage: 147.35 MB
it 1     Memory usage: 1682.64 MB
it 2     Memory usage: 1682.98 MB
it 3     Memory usage: 1682.99 MB
it 4     Memory usage: 1682.99 MB
it 5     Memory usage: 1682.99 MB
it 6     Memory usage: 1678.63 MB
it 7     Memory usage: 1644.47 MB
it 8     Memory usage: 1633.58 MB
it 9     Memory usage: 1633.56 MB

Output (memory leak)

it 0     Memory usage: 108.89 MB
it 1     Memory usage: 1635.26 MB
it 2     Memory usage: 2393.71 MB
it 3     Memory usage: 3156.51 MB
it 4     Memory usage: 3919.34 MB
it 5     Memory usage: 4681.07 MB
it 6     Memory usage: 5428.71 MB
...
MemoryError: Unable to allocate 763. MiB for an array with shape (100000000,) and data type float64

Maybe this answer helps some people to reproduce / solve this Matplotlib memory leak issue. This is probably the 5th time I've come across it.

Upvotes: 1

Related Questions