assertion error with np.load following numpy.savez

Question

I have 5 numpy arrays a,b,c,d and e all defined as:

array([1, 2, 3, 4, 5, 6, 7, 8, 9])

I am saving these arrays like so:

np.savez_compressed('tmp/test',a=a,b=b,c=c,d=d,e=e)

This results in a file, test.npz being created.

However I am having problems when trying to load data in (follows example here):

>>> f=np.load('tmp/test.npz')
>>> f.files
['a', 'c', 'b', 'e', 'd']
>>> f['a']

Gives a large string of errors ending in:

File "C:\Python27\lib\compiler\transformer.py", line 754, in atom_lbrace
return self.com_dictorsetmaker(nodelist[1])
File "C:\Python27\lib\compiler\transformer.py", line 1214, in com_dictorsetmaker
assert nodelist[0] == symbol.dictorsetmaker
AssertionError

I have considered using pickle instead. However this results in file sizes four times that of the .npz files so I'd like to use savez or savez_compressed.

Does anyone have any idea what I'm doing wrong, or suggestions for alternative approaches?

Here is a script that will produce the error:

def saver():
    import numpy as np
    a= np.arange(1,10)
    b=a
    c=a
    d=a
    e=a
    np.savez_compressed('tmp/test',a=a,b=b,c=c,d=d,e=e)
    f=np.load('tmp/test.npz')
    print f.files
    print f['a']

Here is the full traceback:

Traceback (most recent call last):
  File "<pyshell#16>", line 1, in <module>
    saver.saver()
  File "C:\Python27\saver.py", line 14, in saver
    print f['a']
  File "C:\Python27\lib\site-packages\numpy\lib\npyio.py", line 241, in __getitem__
    return format.read_array(value)
  File "C:\Python27\lib\site-packages\numpy\lib\format.py", line 440, in read_array
    shape, fortran_order, dtype = read_array_header_1_0(fp)
  File "C:\Python27\lib\site-packages\numpy\lib\format.py", line 336, in read_array_header_1_0
    d = safe_eval(header)
  File "C:\Python27\lib\site-packages\numpy\lib\utils.py", line 1156, in safe_eval
    ast = compiler.parse(source, mode="eval")
  File "C:\Python27\lib\compiler\transformer.py", line 53, in parse
    return Transformer().parseexpr(buf)
  File "C:\Python27\lib\compiler\transformer.py", line 132, in parseexpr
    return self.transform(parser.expr(text))
  File "C:\Python27\lib\compiler\transformer.py", line 124, in transform
    return self.compile_node(tree)
  File "C:\Python27\lib\compiler\transformer.py", line 159, in compile_node
    return self.eval_input(node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 194, in eval_input
    return Expression(self.com_node(nodelist[0]))
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 578, in testlist
    return self.com_binary(Tuple, nodelist)
  File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary
    return self.lookup_node(n)(n[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 596, in test
    then = self.com_node(nodelist[0])
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 610, in or_test
    return self.com_binary(Or, nodelist)
  File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary
    return self.lookup_node(n)(n[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 615, in and_test
    return self.com_binary(And, nodelist)
  File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary
    return self.lookup_node(n)(n[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 619, in not_test
    result = self.com_node(nodelist[-1])
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 626, in comparison
    node = self.com_node(nodelist[0])
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 659, in expr
    return self.com_binary(Bitor, nodelist)
  File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary
    return self.lookup_node(n)(n[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 663, in xor_expr
    return self.com_binary(Bitxor, nodelist)
  File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary
    return self.lookup_node(n)(n[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 667, in and_expr
    return self.com_binary(Bitand, nodelist)
  File "C:\Python27\lib\compiler\transformer.py", line 1082, in com_binary
    return self.lookup_node(n)(n[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 671, in shift_expr
    node = self.com_node(nodelist[0])
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 683, in arith_expr
    node = self.com_node(nodelist[0])
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 695, in term
    node = self.com_node(nodelist[0])
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 715, in factor
    node = self.lookup_node(nodelist[-1])(nodelist[-1][1:])
  File "C:\Python27\lib\compiler\transformer.py", line 727, in power
    node = self.com_node(nodelist[0])
  File "C:\Python27\lib\compiler\transformer.py", line 805, in com_node
    return self._dispatch[node[0]](node[1:])
  File "C:\Python27\lib\compiler\transformer.py", line 739, in atom
    return self._atom_dispatch[nodelist[0][0]](nodelist)
  File "C:\Python27\lib\compiler\transformer.py", line 754, in atom_lbrace
    return self.com_dictorsetmaker(nodelist[1])
  File "C:\Python27\lib\compiler\transformer.py", line 1214, in com_dictorsetmaker
    assert nodelist[0] == symbol.dictorsetmaker
AssertionError

Answer 1

I do have the same problem(AssertionError) when using numpy 1.7.1/1.8.0 with python 2.7.6 both on MAC OS and Windows. But the problem was automatically fixed after I switch to linux with python 2.7.5. Then I reinstall python 2.7.5 on MACOS and Windows and all the problem was gone. Basically the problem is with python instead of numpy as the compiler is sending alert.So mightly the version matters.

But though npy is the serializable type of numpy, I don't think the file is small enough even with savez_compressed for large matrix.

Hopefully your problem is the same with mine.

Answer 2

Cannot reproduce your problem neither on Linux or Mac (Python 2.7, numpy 1.6.1/1.7.1)

But, I've noticed you using relative path for saving file tmp/test.npz. Is that intentional? In my recollection recent versions of Windows have some special treatment for a new files applications trying to create in some directories (like '/Program Files/') - it moves them away but still tells application they are there in some cases. It does not seems to be likely case here, but can you try absolute path for a file you are saving?

BTW: As alternative to ZIP-archive (which savez savez_compressed creates) you can try pickle with 'LZMAFile' as a file object. It gives very good compression rate (but it can be slow and require more memory and time while compressing/saving file);

It is used as any other file-object wrapper, something like that (to load pickled data):

from lzma import LZMAFile
import cPickle as pickle

if fileName.endswith('.xz'):
    dataFile = LZMAFile(fileName,'r')
else:
    dataFile = file(fileName, 'ro')     
data = pickle.load(dataFile)