How do I save a 3D Python/NumPy array as a text file?

Question

I have to launch a great number of calculations, and I have to save a 2D file text each time, so I would like to store results in "real-time" as a 3D text file with each slice corresponding to one calculation result.

The first calculation is OK, but when I do the second calculation, during the "np.loadtxt" step, the array dimensions become 2D... So I can't reach my aim... and I can't do a reshape when I begin to dimensions (... , ... , 1)

#MY FIRST RESULTS
test1 = open("C:/test.txt", "r")
test_r = np.genfromtxt(test, skip_header=1)
test_r = np.expand_dims(test_r, axis=2) #I create a new axis to save in 3D
test1.close()

#I test if the "Store" file to keep all my results is created.
try:
    Store= np.loadtxt('C:/Store.txt')
except:
    test=1

#If "Store" is not created, I do it or I concatenate in my file.
if test ==1:
    Store= test_r
    np.savetxt('C:/Store.txt', Store)
    test=2
else:
    Store = np.concatenate((Store,test_r), axis=2)
    np.savetxt('C:/Store.txt', Store)


#MY SECOND RESULTS
test2 = open("C:/test.txt", "r")
test_r = np.genfromtxt(test, skip_header=1)
test_r = np.expand_dims(test_r, axis=2)
test2.close()

#I launch the procedure again to "save" the results BUT DURING THE LOADTXT STEP THE ARRAY DIMENSIONS CHANGE TO BECOME A 2D ARRAY...
try:
    Store= np.loadtxt('C:/Store.txt')
except:
    test=1


if test ==1:
    Store= test_r
    np.savetxt('C:/Store.txt', Store)
    test=2
else:
    Store = np.concatenate((Store,test_r), axis=2)
    np.savetxt('C:/Store.txt', Store)

Answer 1

If the save file needs to be a 'csv' style text, you could use multiple savetxt and loadtxt. The key is knowning that both of these can take an open file as input.

Writing example:

In [31]: A=np.arange(3*2*4).reshape(3,2,4)    
In [32]: A    # normal display as 3 blocks of 2d array
Out[32]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

In [36]: for a in A:print a, '\n'   # or iterate on the 1st dimension
[[0 1 2 3]
 [4 5 6 7]] 

[[ 8  9 10 11]
 [12 13 14 15]] 

[[16 17 18 19]
 [20 21 22 23]] 

Following that example, I can iterate on the file, using savetxt for each subarray:

In [37]: with open('3dcsv.txt','wb') as f:
    for a in A:
        np.savetxt(f, a, fmt='%10d')
        f.write('\n')
   ....:         

Confirm the file write with a system cat (via ipython):

In [38]: cat 3dcsv.txt
         0          1          2          3
         4          5          6          7

         8          9         10         11
        12         13         14         15

        16         17         18         19
        20         21         22         23

For simple read, loadtxt apparently ignores the empty lines, returning a 6 x 4 array. So I know that it is supposed to be (2,3,4) I can easily reshape the result.

In [39]: np.loadtxt('3dcsv.txt')
Out[39]: 
array([[  0.,   1.,   2.,   3.],
       [  4.,   5.,   6.,   7.],
       [  8.,   9.,  10.,  11.],
       [ 12.,  13.,  14.,  15.],
       [ 16.,  17.,  18.,  19.],
       [ 20.,  21.,  22.,  23.]])

After a bit of debugging I got this multiple loadtxt to work. loadtxt (and genfromtxt) works with a list of lines.

In [53]: A1=[]     # list to collect blocks

In [54]: with open('3dcsv.txt') as f:
    lines = []     # list to collect lines
    while 1:
        aline = f.readline()
        if aline.strip():
            lines.append(aline)     # nonempty line
        else:              # empty line
            if len(lines)==0: break
            A1.append(np.loadtxt(lines, dtype=int))
            lines = []
   ....:             

In [55]: A1 = np.array(A1)

In [56]: A1
Out[56]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

This may not be the most robust pairing of of save/load but it gives a framework for building something better.

But if it doesn't need to be text then pickle is fine, as is the native numpy 'save/load'

In [57]: np.save('3dsave.npy',A)

In [58]: np.load('3dsave.npy')
Out[58]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

Answer 2

Here's an example of cPickle:

import cPickle

# write to cPickle
cPickle.dump( thing_to_save, open( "filename.pkl", "wb" ) )

# read from cPickle
thing_to_save = cPickle.load( open( "filename.pkl", "rb" ) )

The "wb" and "rb" parameters to the open() function are important. CPickle writes objects in a binary format, so using just "w" and "r" won't work.