CODEWITHSUNDEEP
pythonnumpy
P. Aumann
P. Aumann

Reputation: 41

Error when loading a Python 2 .npy file in Python 3

I have .npy files that were created using Python 2.7.9 and Numpy Version 1.11.3 with the command np.save('filename'). The files were produced on an external machine that is part of the linux-cluster of our institute. I copied the files to my local machine in order to import them via np.load('filename.npy'). On my local machine I am running Python 3.5.2 and Numpy Version 1.13.0 with Jupyter-Notebook. The local OS is Ubuntu 16.04.2.

When I try to load the files locally I get the error:

ValueError: invalid literal for int() with base 16

After browsing through some Stackoverflow questions I tried to specify the encoding with: 

np.load('filename.npy',encoding='latin1')

This gives the same error. encoding='bytes' yields:

TypeError: can't multiply sequence by non-int of type 'float'

Here is a larger snippet of the Traceback:

/usr/local/lib/python3.5/dist-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
417             else:
418                 return format.read_array(fid, allow_pickle=allow_pickle,
--> 419                                          pickle_kwargs=pickle_kwargs)
420         else:
421             # Try a pickle

/usr/local/lib/python3.5/dist-packages/numpy/lib/format.py in read_array(fp, allow_pickle, pickle_kwargs)
638             pickle_kwargs = {}
639         try:
--> 640             array = pickle.load(fp, **pickle_kwargs)
641         except UnicodeError as err:
642             if sys.version_info[0] >= 3:

/usr/local/lib/python3.5/dist-packages/sympy/core/numbers.py in __new__(cls, num, prec)
823                 else:
824                     _mpf_ = mpmath.mpf(
--> 825                         S.NegativeOne**num[0]*num[1]*2**num[2])._mpf_
826         elif isinstance(num, Float):
827             _mpf_ = num._mpf_

TypeError: can't multiply sequence by non-int of type 'float'

I guess that something with the encoding went wrong on the transition between the Python and Numpy versions. Any ideas on how I can import the files?

Upvotes: 2

Views: 2310

Answers (1)

cat
cat

Reputation: 4020

As shown in What is the way data is stored in *.npy?, .npy files are bytecode, which you will see if you open one in a hex editor.

Python 2 bytecode .pyc, .pyo files cannot be run in Python 3, as the virtual machine and compiler internals have changed with the major version.

Similarly, NumPy's C internals and bytecode compiler have changed as well in Python 3, breaking backwards compatibility. (This is intentional since bytecode is not meant to persist for so long, or be used in different versions than it was created.)

The composition of these changes means that there is no way, without huge changes to Python 3's bytecode interpreter and Python 3's NumPy, and/or a transpiler from Python 2 NumPy bytecode to that of Python 3, to use these Python 2 .npy files in Python 3.

As I alluded to earlier, this is a bit of an X/Y Problem. You should not be reliant on the .npy files to work across versions, because it is not guaranteed they will as they are inherently a volatile format (like Python VM bytecode).

Instead of reverse-engineering the bytecode to debug it, try to get the source from which these files were generated.

Upvotes: 2

Related Questions