CODEWITHSUNDEEP
pythonnumpyfile-io
FJC
FJC

Reputation: 166

How to append 3D numpy array to file?

I am trying to save a 100x100x100 array of integers to a file with the date and time it was saved as a header. It doesn't need to be human-readable (except for the time-stamp header) so I was planning to use numpy.save(), take one slice at a time and save it to the file, but this does not append to the end of the file, it overwrites each time so the file only ends up containing the last slice.

Is there something like save() or savetxt() which appends to a file rather that overwrites?

Note: if it makes it easier, could I put the date/time into the filename when it saves instead of into the header?

My current attempt looks something like this:

with open("outfile.txt",'w') as mfile:
    mfile.write(strftime("%x %X\n"))
for i in range(len(x)):
    np.savetxt("outfile.txt",x[i])

Upvotes: 0

Views: 824

Answers (3)

Aaron
Aaron

Reputation: 11075

I'm a fan of Pickles :)

import cPickle
import time
import numpy as np

arr = np.array(xrange(100)).reshape(10,10)

#write pickle file
with open('out.p', 'wb') as f:
    t = time.asctime()
    cPickle.dump(t, f, cPickle.HIGHEST_PROTOCOL)
    cPickle.dump(arr, f, cPickle.HIGHEST_PROTOCOL)

#read pickle file
with open('out.p', 'rb') as f:
    t = cPickle.load(f)
    arr = cPickle.load(f)

Upvotes: 1

galath
galath

Reputation: 5965

Use the 'a'flag to append to a file. numpy.savetxt takes an array structure as input, so we need to reshape it.

p,q,r = x.shape
with open("outfile.txt",'ab') as mfile:
    header = strftime("%x %X\n")
    np.savetxt(mfile, x.reshape(p*q*r), header=header)

Upvotes: 1

FJC
FJC

Reputation: 166

With the help of @galath's advice (although actually it doesn't seem to matter if I use 'a' or 'w' flag now...), I have figured out the following method using np.save/load and having the date as a header:

outfile="out.npy"

with open(outfile,'w') as mfile:
    mfile.write(strftime("%x %X\n"))      #write date and time as header
    for i in range(len(x)):
        np.save(mfile,x[i])               #save each slice

readx=np.zeros((len(x),len(y),len(z)) #empty array to be filled from file    

with open(outfile,'r') as mf:                #reopen file (for reading in)
    dt=mf.readline()                      #read date/time header
    for i in range(len(x)):
        readx[i,:,:]=np.load(mf)          #read in each slice and save it to array

If anyone has a more elegant solution feel free to share, but this way does what I need for now.

Upvotes: 0

Related Questions