CODEWITHSUNDEEP
pythonnumpy
DavidC.
DavidC.

Reputation: 677

Why np.arrays and lists work with `allow_pickle=False`, but dictionaries do not?

This question involves np.save and np.load best practices. Since the newer numpy version 1.16.3, the default in np.load is set to allow_pickle=False.

After saving a list, the further load declaration works just fine with the default allow_pickle=False:

>> x = [0, 1, 2]                            
>> np.save('my_x_list.npy', x)                
>> loaded_x = np.load('my_x_list.npy') 
>> loaded_x                        
Out: array([0, 1, 2])

The same holds for a numpy array:

>> y = np.arange(10)                            
>> np.save('my_y_numpy_array.npy', y)                
>> loaded_y = np.load('my_y_numpy_array.npy') 
>> loaded_y                          
Out: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

However, a dictionary yields this error:

>> mydict = {'a': 4, 'b': 5}
>> np.save('my_dict.npy', mydict)
>> loaded_z = np.load('my_z_dict.npy')
ValueError: Object arrays cannot be loaded when allow_pickle=False

As far as I understand, dictionaries, lists and numpy arrays are all Object arrays. Hence, one would expect numpy arrays or lists to raise this error as well. Why is this error raised with dictionaries and is not raised with numpy arrays or lists ?

Upvotes: 1

Views: 1570

Answers (1)

michaeldel
michaeldel

Reputation: 2385

As far as I understand, dictionaries, lists and numpy arrays are all Object arrays

No, it depends on the data type of the values in the list. The reason why you are encountering this error is because you are trying to create a numpy.array from a dict object, which will always give an “Object Array”, that is to say a numpy.array with dtype=object. See

>>> import numpy as np
>>> np.array({'a': 4, 'b': 5})
array({'a': 4, 'b': 5}, dtype=object)

Whereas, when using a list of numbers (integers, floats, complex numbers, etc.) to create a numpy.array, that array will have a number dtype, that does not require to be pickled here.

>>> np.array([1, 2, 3]).dtype
dtype('int64')

You can load a dictionary (or even other objects) from a file into a numpy.array, using the allow_pickle parameter, e.g.

np.load('dictionary.npy', allow_pickle=True)

Upvotes: 3

Related Questions