CODEWITHSUNDEEP
pythonarraysnumpy
Chris
Chris

Reputation: 31256

Saving List of Numpy 2D arrays using numpy.save (the arrays together are jagged)

I have a large image dataset. When I use the images, I have several components--a mirrored image, a regular image, an eigenvector matrix and an eigenvalue vector.

I would like to store it like: 

training_sunsets_data = [cropped_training_sunsets,
                         mirrored_training_sunsets,
                         rgb_cov_eigvec_training_sunsets,
                         rgb_cov_eigval_training_sunsets]

np.save('training_sunsets_data',training_sunsets_data)

And as I was writing this I was testing it (because I was sure it would fail), and the strangest thing happened when I did this: it worked.

Further, when I loaded it back up into the code, it was type ndarray, but it is a jagged array.

How is this possible if numpy does not allow jagged multidimensional arrays? Did I just find a backdoor way to create a jagged array in numpy?

Upvotes: 13

Views: 25105

Answers (3)

Dan Erez
Dan Erez

Reputation: 1504

np.savez() would work in your situation. save each as a variable.

Upvotes: 2

gdlmx
gdlmx

Reputation: 6789

After testing on my machine:

  import numpy as np
  np.save('testnp.npy', [[2,3,4],[1,2]])
  np.load('testnp.npy')
  #   array([[2, 3, 4], [1, 2]], dtype=object)

As shown in the example code, the loaded object is of type ndarray, but its data type is object. That means, np.save store an array of python objects, which can be anything. According to the documentation, it seems to use python pickle to pack those objects.

So you didn't find a backdoor, it behaves just as expected.

Upvotes: 13

kpie
kpie

Reputation: 11110

So to see what you are getting at lets runs some code.

>>> a =[np.array([[1,2,3],[4,5,6]]),np.array([[1,2],[3,4]])]
>>> type(a)
<type 'list'>
>>> np.array(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: could not broadcast input array from shape (2,3) into shape (2)

We see here that we are perfectly able to make a list of np.arrays of different dimensions. We cannot however cast that list into a np.array.

I suspect based on your syntax that you are saving a list, and loading a list maintaining the type np.array for each element in the list.

Upvotes: 0

Related Questions