CODEWITHSUNDEEP
pythonmemory-managementmatplotlibpyodbc
jaromey
jaromey

Reputation: 676

Efficient memory usage in python

I wrote a Python script using pyodbc to transfer data from an excel sheet into ms access and also using matplotlib to use some of the data in the excel sheet to create plots and save them to a folder. When I ran the script, it did what I expected it to do ; however, I was monitoring it with task manager and it ended up using over 1500 MB of RAM!

I don't even understand how that is possible. It created 560 images but the total size of those images was only 17 MB. The excel sheet is 8.5 MB. I understand that maybe you can't tell me exactly what the problem is without seeing all my code (I don't know exactly what the problem is so I would just have to post the whole thing and I don't think it is reasonable to ask you to read my entire code) but some general guidelines would suffice.

Thanks.

Update

I did just as @HYRY suggested and split my code up. I ran the script first with only the matplotlib functions and then afterwards without them. As those who have commented so far have suspected, the memory hog is from the matplotlib functions. Now that we have narrowed it down I will post some of my code. Note that the code below executes in two for loops. The inner for loop will always execute four times while the outer for loop executes however many times is necessary.

#Plot waveform and then relative harmonic orders on a bar graph.
#Remember that table is the sheet name which is named after the ExperimentID
cursorEx.execute('select ['+phase+' Time] from ['+table+']')
Time = cursorEx.fetchall()                   
cursorEx.execute('select ['+phase+' Waveform] from ['+table+']')
Current = np.asanyarray(cursorEx.fetchall())                                                               
experiment = table[ :-1]                
plt.figure()
#A scale needs to be added to the primary current values
if line == 'P':
    ratioCurrent = Current / 62.5
    plt.plot(Time, ratioCurrent)
else:
    plt.plot(Time, Current)
plt.title(phaseTitle)
plt.xlabel('Time (s)')
plt.ylabel('Current (A)')
plt.savefig(os.getcwd()+'\\HarmonicsGraph\\'+line+'H'+experiment+'.png')

cursorEx.execute('select ['+phase+' Order] from ['+table+']')
cursorEx.fetchone() #the first row is zero
order = cursorEx.fetchmany(51)
cursorEx.execute('select ['+phase+' Harmonics] from ['+table+']')
cursorEx.fetchone()
percentage = np.asanyarray(cursorEx.fetchmany(51))

intOrder = np.arange(1, len(order) + 1, 1)  
plt.figure()                
plt.bar(intOrder, percentage, width = 0.35, color = 'g')                
plt.title(orderTitle)
plt.xlabel('Harmonic Order')
plt.ylabel('Percentage')
plt.axis([1, 51, 0, 100])
plt.savefig(os.getcwd()+'\\HarmonicsGraph\\'+line+'O'+experiment+'.png')

Upvotes: 1

Views: 546

Answers (2)

Pablo
Pablo

Reputation: 2481

I think that plt.close() is a very hard solution when you will do more than one plot in your script. A lot of times, if you keep the figure reference, you can do all the work on them, calling before:

plt.clf()

you will see how your code is faster (is not the same to generate the canvas every time!). Memory leaks are terrible when you call multiple axes o figures without the proper clean!

Upvotes: 2

Brad
Brad

Reputation: 1367

I don't see a cleanup portion in your code, but I'd be willing to bet that the issue is that you are not calling

plt.close()

after you are finished with each plot. Add that one line after you are done with each figure and see if it helps.

Upvotes: 1

Related Questions