matplotlib generate animation, each file is a new frame

Question

First, I'd like to thank anyone that can get me through this in advance. I'm new to Matplotlib, and generally don't code in python either.

What I have is a massive number of data files (100's-10,000's). Each of those files have 20 plots I'd like to turn into an animation, where each file represents a different frame. The code has become incredibly convoluted in trying to get things to work. There are 6 subplots (3,2,1-6). They all share the same x axis. On any given subplot, I have 1 to 6 y values plotted against it. They also need the proper labels and some are 'symlog' plots as I would like to view logarithmic data in both positive and negative. I would like the animation to be formed without relying on ffmpeg or mencoder as those may not be available on the computers running the code. It seems the obvious solution to this lies in FuncAnimation, but that really just confuses me way too much. Most examples I've seen simply add another point to a single plot. I want to replace the data in basically 20 plots. I'll include just two subplots and assume I'm smart enough to extrapolate that to 6...

I've successfully gotten plots to form as .png files in a separate "safe" file.

So here's some really, really, ugly code:

#!/usr/bin/python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import matplotlib.animation as animation
import glob
import os
import re
from StringIO import StringIO
from matplotlib.ticker import MaxNLocator
from matplotlib.font_manager import FontProperties

def GetFiles(dir_name, extension):
    fileList = []
    for file in os.listdir(dir_name):
        index = file.find(extension)
        if index != -1:
            dirfile = os.path.join(dir_name, file)
            fileList.append(dirfile)
    return fileList

dirString = raw_input('Please enter a directory name: ')
extension = raw_input('Please enter the extension: ')
ClearFiles = raw_input('Remove temporary .png files (Y/N): ')
dataName = GetFiles(dirString, extension)
print dataName  #just make sure right files are being loaded
pngfilelist = []

figure = plt.figure()
fontP = FontProperties()
fontP.set_size('small')

ax1 = figure.add_subplot(1,2,1)
ax1.grid(True)  # Enable Grid Lines
ax1.legend(loc=9, ncol=6, borderaxespad=0., prop=fontP)
ax1.set_title('Band Diagram')
PsiPlot, = ax1.plot([], [], label='Psi')
EcPlot, = ax1.plot([], [], label='Ec')
EvPlot, = ax1.plot([], [], label='Ev')
ax1.legend(loc=9,ncol=6, borderaxespad=0., prop=fontP)
ax2 = figure.add_subplot(1,2,2, sharex=ax1)
NPlot, = ax2.plot([], [], label='n')
PPLot, = ax2.plot([], [], label='p')
ax2.grid(True)  # Enable Grid Lines
ax2.set_title('Electron/Hole Concentrations')
ax2.legend(loc=9,ncol=2, borderaxespad=0., prop=fontP)
X = []
Psi = []
Ec = []
Ev = []
n = []
p = []


def UpdatePlot(dataFile):
    data = np.genfromtxt(dataFile, autostrip=True, skip_header=4, names=True, usecols=("X", "Psi", "Ec", "Ev", "n", "p"))  #Load the specified data into giant list

    entries = len(data)
    for ctr in range(0,entries):
        X.append(data[ctr][0])  # X-value for all plots
        Psi.append(data[ctr][1])    #plot 1,1
        Ec.append(data[ctr][2])
        Ev.append(data[ctr][3])
        n.append(data[ctr][4]) # plot 1,2
        p.append(data[ctr][5])

    figure.suptitle(dataFile, y=0.99)
    PsiPlot.set_data(X, Psi)
    EcPlot.set_data(X, Ec)
    EvPlot.set_data(X, Ev)
    NPlot.set_data(X, n)
    PPlot.set_data(X, p)

    plt.subplot(1,2,1)
    plt.xlabel('Position (cm)')
    plt.ylabel('Energy (eV)')

    plt.subplot(1,2,2)
    plt.xlabel('Position (cm)')
    plt.ylabel('cm^-3')
    plt.yscale('symlog', linthreshy=1e10)

    figure.set_size_inches(16,10)
    figure.set_dpi(200)

    plt.tight_layout(pad=0.2, w_pad=0.2, h_pad=0.2)
    filename = dataFile.replace(extension, '.png')
    plt.savefig(filename)
    pngfilelist.append(filename)

    return PsiPlot, EcPlot, EvPlot, NPlot, PPlot,

ani = animation.FuncAnimation(figure, UpdatePlot, dataName, interval=500, blit=True)

ani.save('test.mp4', fps=15)

# remove the temporary png files if wanted.
if ClearFiles == 'y' or ClearFiles == 'Y':
    for fname in pngfilelist:
        os.remove(fname)

Things I realize: The append data is not the nicest way to get all the X and Y data in lists. The 2nd set of data will also include the first as written in this snippet (looking for good way to delete it that doesn't cause problems later down the line). There are probably a lot more imports than I currently need as I tried various things, and then cut them out. Using this method, the X/Y scales are not automatically set as it is when all I try to do is save to a .png file (I do everything via plt.plot and clear after saving instead of all this stuff about setting data to an axis). For instance, I'd like to set the lowest Y value by the lowest Ev and the highest Y value by the highest Psi. Also, nothing seems to work on the 2nd plot. Granted, the values are very large.

With this code, I'm getting a warning of "No labeled objects found." and a runtime error saying Underlying C/C++ object has been deleted - two errors I don't receive in the just plot and save to .png file code.

I'm really just at a loss as to how to get all these graphs into FuncAnimation, though.

Any thoughts? I'm sick of pounding my head on Python - I really need to be pounding my head on my simulation code.

Finally, here's a sample portion of an old (bad) data file:

Data from: TestLoad.dev
Simulation time: 4.08738e-013
Iteration : 665
Data binning: 5
Point          X            Psi             Ec             Ev          Fermi        Fermi_n            Efp              n              p            n*p            Rho    Ec_Carriers    Ev_Carriers      Light_Gen  Generation_Th Recomb_Thermal      SRH_Rate1      SRH_Rate2      SRH_Rate3      SRH_Rate4           Jn_x           Jp_x
0       4.9e-006           3.58      -0.500001      -0.500001           -0.5           -0.5      -0.500001   2.72507e+021   2.72507e+021   7.42603e+042              0   2.67057e+008   2.67057e+008              0              0              0              0              0              0              0        4577.65              0
1      9.95e-006           3.58           -0.5           -0.5           -0.5      -0.499999           -0.5   2.72508e+021   2.72508e+021   7.42603e+042              0   8.17523e+006   8.17523e+006              0              0              0              0              0              0              0              0        -114441
2     1.015e-005        3.61356      0.0255559       -1.09444       -0.95916       -0.95916      -0.830208              0   1.08799e+015              0      -0.132665              0       0.971429              0              0              0              0              0              0              0              0        -114441
3     1.025e-005        3.62841      0.0404132       -1.07959      -0.944094      -0.944094      -0.844848              0   2.89096e+015              0      -0.132656              0        3.02857              0              0              0              0              0              0              0              0        -119019
4     1.035e-005        3.64199      0.0539899       -1.06601      -0.930857      -0.930857      -0.854293              0   9.46844e+015              0      -0.131488              0        10.3143              0              0              0              0              0              0              0              0        -114441
5     1.045e-005         3.6543      0.0662974        -1.0537      -0.919519      -0.919519      -0.867723              0   2.36441e+016              0      -0.129511              0        21.6571              0              0              0              0              0              0              0              0        -123596
6     1.055e-005        3.66535      0.0773546       -1.04265      -0.909748      -0.909748      -0.873209              0   4.47623e+016              0      -0.125061              0        48.4286              0              0              0              0              0              0              0              0       -96130.6
7     1.065e-005         3.6752      0.0872047        -1.0328      -0.901449      -0.901449      -0.876584              0    6.9861e+016              0        -0.1222              0        66.2857              0              0              0              0              0              0              0              0        -146485
8     1.075e-005        3.68388      0.0958752       -1.02412      -0.895041      -0.895041      -0.880708              0   1.18029e+017              0      -0.113068              0        124.286              0              0              0              0              0              0              0              0       -86975.3
9     1.085e-005        3.69145       0.103454       -1.01655      -0.889233      -0.889233      -0.879943              0   1.57625e+017              0      -0.111058              0        136.829              0              0              0              0              0              0              0              0        -137329
10     1.095e-005        3.69796       0.109961       -1.01004      -0.885743      -0.885743      -0.881837              0   2.16895e+017              0     -0.0941347              0        240.457              0              0              0              0              0              0              0              0       -22888.2
788     0.00998975        4.19373       0.605734      -0.514266        -0.3792        -0.3792      -0.287991              0   5.78298e+015              0      -0.131942              0        5.48571              0              0              0              0              0              0              0              0          77820
789     0.00998985        4.17975       0.591751      -0.528249      -0.393181      -0.393181      -0.292558              0   2.27746e+015              0      -0.132404              0            1.6              0              0              0              0              0              0              0              0        68664.7
790      0.0099904           4.08  -1.45745e-006  -1.45745e-006   3.16863e-016   4.06816e-008  -7.67735e-007   2.72507e+021   2.72507e+021   7.42603e+042              0   2.72507e+007   2.72507e+007              0              0              0              0              0              0              0              0              0
791     0.00999545           4.08  -1.45745e-006  -1.45745e-006   3.16863e-016   4.06816e-008  -7.67735e-007   2.72507e+021   2.72507e+021   7.42603e+042              0   2.47982e+008   2.47982e+008              0   6.27943e+027              0              0              0              0              0              0              0

---

Just in case future people look at this, here's what ended up being my final, simplified answer (proof of concept) before I start throwing in all the other plots.

#!/usr/bin/python
import numpy as np
import matplotlib as mpl
mpl.use( "agg" )
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import os
from StringIO import StringIO
from matplotlib.font_manager import FontProperties

def GetFiles(dir_name, extension):
    fileList = []
    for file in os.listdir(dir_name):
        index = file.find(extension)
        if index != -1:
            dirfile = os.path.join(dir_name, file)
            fileList.append(dirfile)
    return fileList

def UpdatePlot(dataFile):
    global MinX, MaxX, ax1MinY, ax1MaxY, ax2MinY, ax2MaxY, first
    print 'loading data for ', dataFile
    data = np.genfromtxt(dataFile, autostrip=True, skip_header=4, names=True, usecols=("X", "Psi", "Ec", "Ev", "n", "p"))  #Load the specified data into giant list

    # get new bounds on limits for graphs

    tempMin = data['X'].min()
    tempMax = data['X'].max()
    if tempMax > MaxX or first == True:
        MaxX = tempMax + (tempMax - tempMin) * 0.01
        ax1.set_xlim(MinX, MaxX)
        ax2.set_xlim(MinX, MaxX)
    if tempMin < MinX or first == True:
        MinX = tempMin - (tempMax - tempMin) * 0.01
        ax1.set_xlim(MinX, MaxX)
        ax2.set_xlim(MinX, MaxX)

    tempMin = data['Psi'].min()
    tempMax = data['Psi'].max()
    if tempMax > ax1MaxY or first == True:
        ax1MaxY = tempMax + (tempMax - tempMin) * 0.5
        ax1.set_ylim(ax1MinY, ax1MaxY)

    tempMin = data['Ev'].min()
    tempMax = data['Ev'].max()
    if tempMin < ax1MinY or first == True:
        ax1MinY = tempMin - (tempMax - tempMin) * 0.2
        ax1.set_ylim(ax1MinY, ax1MaxY)

    tempMax = data['n'].max()
    if tempMax > ax1MaxY or first == True:
        ax2MaxY = tempMax * 2    # This is basically a log plot...
        ax2.set_ylim(ax2MinY, ax2MaxY)

    tempMax = data['p'].max()
    if tempMax > ax1MaxY or first == True:
        ax2MaxY = tempMax * 2    # This is basically a log plot...
        ax2.set_ylim(ax2MinY, ax2MaxY)

    # Now update all the data for the plots

    titleText.set_text(dataFile)
    PsiPlot.set_data(data['X'], data['Psi'])
    EcPlot.set_data(data['X'], data['Ec'])
    EvPlot.set_data(data['X'], data['Ev'])
    NPlot.set_data(data['X'], data['n'])
    PPlot.set_data(data['X'], data['p'])

    plt.draw()  # need to update the figure regardless because the title changes
    if GeneratePNG == 'Y' or GeneratePNG == 'y':
        filename = dataFile.replace(extension, '.png')
        plt.savefig(filename)

    first = False

    return

dirString = raw_input('Please enter a directory name: ')
extension = raw_input('Please enter the extension: ')
GeneratePNG = raw_input('Generate .png files of each file (Y/N): ')
framesps = raw_input('Frames per second: ')
outname = raw_input('Output file name (no extension): ')
outname = outname + '.mp4'
dataName = GetFiles(dirString, extension)
# print dataName

MinX = 0
MaxX = 0
ax1MinY = 0
ax1MaxY = 0
ax2MinY = 0
ax2MaxY = 0
first = True

figure = plt.figure()
fontP = FontProperties()
fontP.set_size('small')
figure.set_size_inches(16,10)
figure.set_dpi(200)
titleText = figure.suptitle('Title', y=0.99)  # must do this way to allow title to be changed later

ax1 = figure.add_subplot(1,2,1)
ax1.grid(True)  # Enable Grid Lines
ax1.set_title('Band Diagram')
plt.xlabel('Position (cm)')
plt.ylabel('Energy (eV)')
PsiPlot, = ax1.plot([], [], label='Psi')
EcPlot, = ax1.plot([], [], label='Ec')
EvPlot, = ax1.plot([], [], label='Ev')
ax1.legend(loc=9,ncol=6, borderaxespad=0., prop=fontP)

ax2 = figure.add_subplot(1,2,2, sharex=ax1)
plt.xlabel('Position (cm)')
plt.ylabel('cm^-3')
plt.yscale('symlog', linthreshy=1e10)
ax2.grid(True)  # Enable Grid Lines
ax2.set_title('Electron/Hole Concentrations')
NPlot, = ax2.plot([], [], label='n')
PPlot, = ax2.plot([], [], label='p')
ax2.legend(loc=9,ncol=2, borderaxespad=0., prop=fontP)

plt.tight_layout(pad=0.5, w_pad=0.5, h_pad=0.5)

ani = mpl.animation.FuncAnimation(figure, UpdatePlot, dataName, init_func=None, interval=500, blit=True)

ani.save(outname, fps=framesps, codec='mpeg4')

Thanks again for pointing me in the right direction!

Answer 1

Ok, I've delved a bit more into your code. There are a few things I can see.

First of all, always try and provide a full backtrace if an error occurs. In this case, the error was reasonably well googleable(*), and lead me to e.g. RuntimeError: underlying C/C++ object has been deleted when saving and afterwards closing a pyplot figure. It's indicative of a problem with pyqt; apparently, that is your default backend.

So, as the answer on that page suggests, try changing your backend:

import matplotlib as mpl
mpl.use( "agg" )

Since you're just saving the files anyway, you don't need an interactive backend anyway. "agg" is a non-interactive one, so no pop-ups with plots, you can only save to disk. That may solve the RuntimeError. I haven't really bothered with the labeling error, though I guess that you're trying to put labels on plots where the actual data are not there. Perhaps this is the unused 6th plot (because I don't see that 6th plot anywhere)?

On reading the data: it seems you now append the next set of data to the current data. That is correct? Or did you want to replace it? If replacing, simply do something like:

Psi = data['Psi']

etc. Without the for-loop of course. Or even simpler, only:

PsiPlot.set_data(data['X'], data['Psi'])

If you are appending, it's a bit trickier, but you can do the following I guess. First, the initial assignments need to be numpy arrays (showing only for Psi again):

Psi = nparray([])

Then, appending becomes:

Psi = np.concatenate((Psi, data['Psi']))

(Note the double set of parenthesis. Again, without the for loop).

I noticed that you get a UnboundLocalError this way. It looks like default Python objects (lists such as []) are automatically found inside a function, but a namespaced object like a numpy.ndarray (which Psi above is), is not. Therefore, at the start of your UpdatePlot, you need to have:

global X, Psi, Ec, Ev, n, p

So that Python finds the right objects to append to.

Lastly, you mentioned you don't want an mencoder or ffmpeg dependency, but under the hood, matplotlib.animation.save() calls these programs. E.g., I tried to run your script and got a RuntimeError, because ffmpeg was not installed on my system. So be aware of this when trying to run this code elsewhere. I don't know of a Python-only mpeg encoder.

(*) In fact, probably so could you. I guess the fact that you got too frustrated by the whole process and Python script meant you didn't split the question up in the error and other details, because you mentioned several issues. It's probably ok, but as usual, it distracts from the actual problem.