Numpy broadcast shape error: convert list( lists(arrays) ) to array

Question

I have a list of lists like list_ABC = [[A,B,C], [A,B,C], ...]
with 2D ndarrays A (2x2), B (2x3) and C (2x3).

Now, I'd like to convert the main list to a numpy array:

np.array(list_ABC)

However, I get the following error:

ValueError: could not broadcast input array from shape (2,2) into shape (2)

I need this conversion because I'd like to get

A_matrices = np.array(list_ABC)[:, 0]
B_matrices = np.array(list_ABC)[:, 1]

Such that I can finally obtain a ndarray containing all A-arrays (array(A,A,A,...)).

Unfortunately I can't get a clue from the value error message. Interestingly, if I only transpose matrix C with C.T (making it a 3x2 matrix) no error is thrown.

Now, I could solve the problem by creating a list_A, list_B, list_C beforehand (and not list_ABC), but this doesn't feel as simple (constructing and appending to each list_A/B/C requires a few more lines of code). Similarly I could use other methods (e.g. using a dict with A,B,C keys containing a list of all A/B/C matrices), but nothing feels so simple like this solution.

A working example which throws the error:

import numpy as np
list = [[np.array([[ 476.,  667.], [ 474.,  502.]]), np.array([[ 343.,  351.,  449.], [ 352.,  332.,  292.]]), np.array([[ 328.,  328.,  294.], [ 367.,  355.,  447.]])], [np.array([[ 497.,  546.], [ 456.,  517.]]), np.array([[ 361.,  342.,  340.], [ 341.,  304.,  328.]]), np.array([[ 347.,  313.,  293.], [ 355.,  333.,  375.]])]]
np.array(list)

Thanks a lot!

Answer 1

When constructing an array from arrays, np.array function can do 3 things:

• if all subarrays have the same shape it will make a higher dimensional array

• if the subarrays differ in shape it may construct an object dtype array. This is like a list, or nested list, but with the ability to index and reshape like arrays

• raise an error. This seems to happen most when rows match but columns don't. It may be detecting the shape mismatch too late to fall back on the object dtype solution.

More on this at:

How to keep numpy from broadcasting when creating an object array of different shaped arrays

---

To get array(A,A,A,...) I would suggest use list comprehension. Constructing object dtype arrays is too tricky.

---

The reliable way of creating an object array is to initialize an empty one and fill it:

In [116]: arr = np.empty((2,3), dtype=object)
In [117]: arr[...] = alist
In [118]: arr
Out[118]: 
array([[array([[ 476.,  667.],
       [ 474.,  502.]]),
        array([[ 343.,  351.,  449.],
       [ 352.,  332.,  292.]]),
        array([[ 328.,  328.,  294.],
       [ 367.,  355.,  447.]])],
       [array([[ 497.,  546.],
       [ 456.,  517.]]),
        array([[ 361.,  342.,  340.],
       [ 341.,  304.,  328.]]),
        array([[ 347.,  313.,  293.],
       [ 355.,  333.,  375.]])]], dtype=object)

Now I can select the 'A' elements:

In [119]: arr[:,0]
Out[119]: 
array([array([[ 476.,  667.],
       [ 474.,  502.]]),
       array([[ 497.,  546.],
       [ 456.,  517.]])], dtype=object)

but that's an object array, and wrapping again in np.array doesn't change that:

In [120]: np.array(arr[:,0])
Out[120]: 
array([array([[ 476.,  667.],
       [ 474.,  502.]]),
       array([[ 497.,  546.],
       [ 456.,  517.]])], dtype=object)

but they can be concatenated on several different axes.

In [121]: np.stack(arr[:,0])
Out[121]: 
array([[[ 476.,  667.],
        [ 474.,  502.]],

       [[ 497.,  546.],
        [ 456.,  517.]]])

But I can get the same thing without the object array step

In [123]: np.stack([a[0] for a in alist])
Out[123]: 
array([[[ 476.,  667.],
        [ 474.,  502.]],

       [[ 497.,  546.],
        [ 456.,  517.]]])