CODEWITHSUNDEEP
pythonpython-3.xlistnumpy
rpb
rpb

Reputation: 3299

Efficient approach to transform list of numpy array in a list

I have a list with two ndarray of shape (1,3) inside the list. There are 3 similar copy of this inner list.Each of these inner list is further grouped in the outer list which correspond to a an outer-list with len 3.

The idea is to transform the nested list from shape 3 (outer) and 2 (inner) into the shape of shape 2 (outer) and 3 (inner).

The following code is draft

from operator import itemgetter

all_l=[[np.array([[1, 2, 3]]),np.array([[4, 5, 6]])],
       [np.array([[1, 2, 3]]),np.array([[4, 5, 6]])],
       [np.array([[1, 2, 3]]),np.array([[4, 5, 6]])]]


trans_format=[list ( map ( itemgetter ( i),all_l) ) for i in range (0,len(all_l[0]))]

But, I wonder whether there more efficient way of achieving this? As in actual implementation, the outer and inner len of the list if huge.

Upvotes: 0

Views: 161

Answers (3)

s3dev
s3dev

Reputation: 9701

You are correct in thinking some efficiency can be gained. Here are some timing comparisons.

It's worth noting that these timings were run on the original dataset, and might show an efficiency shift when scaled up. (Work in progress.)

Answer by @fountainhead:
Time per loop: 10.1 µs

new_arr = np.array(all_l).transpose(1,0,2,3)

Original:
Time per loop: 3.44 µs

trans_format=[list ( map ( itemgetter ( i),all_l) ) for i in range (0,len(all_l[0]))]

Answer by @rapchen:
Time per loop: 1.44 µs

trans_format = [list(row) for row in zip(*all_l)]

This answer (verbose/basic approach):
Time per loop: 1.18 µs

l1 = []
l2 = []
for i in l:
    l1.append(i[0])
    l2.append(i[1])
    
trans_format = [l1, l2]

Output for all answers:
(Answer by @foundtainhead outputs a numpy array.)

[[array([[1, 2, 3]]), array([[1, 2, 3]]), array([[1, 2, 3]])],
 [array([[4, 5, 6]]), array([[4, 5, 6]]), array([[4, 5, 6]])]]

Upvotes: 1

fountainhead
fountainhead

Reputation: 3722

The following code will give your desired data-structure as a numpy array:

new_arr = np.array(my_list).transpose(1,0,2,3)

Testing it out:

from pprint import pprint
my_list = [[np.arange(3).reshape(1,3), np.arange(3,6).reshape(1,3)],
           [10* np.arange(3).reshape(1,3), 10* np.arange(3,6).reshape(1,3)],
           [100*np.arange(3).reshape(1,3), 100*np.arange(3,6).reshape(1,3)]]
pprint (my_list)
new_arr = np.array(my_list).transpose(1,0,2,3)
print (new_arr.shape)
print (new_arr.dtype)
print (new_arr)

Output:

[[array([[0, 1, 2]]), array([[3, 4, 5]])],
 [array([[ 0, 10, 20]]), array([[30, 40, 50]])],
 [array([[  0, 100, 200]]), array([[300, 400, 500]])]]

(3, 2, 1, 3)
int32
[[[[  0   1   2]]

  [[  3   4   5]]]


 [[[  0  10  20]]

  [[ 30  40  50]]]


 [[[  0 100 200]]

  [[300 400 500]]]]

Converting back to 2x3 list of numpy arrays:

(Can't imagine why you'd want to do that !)

new_list = [list(slice) for slice in new_arr]

Testing the conversion:

pprint (new_list)

Output of the conversion:

[[array([[0, 1, 2]]), array([[ 0, 10, 20]]), array([[  0, 100, 200]])],
 [array([[3, 4, 5]]), array([[30, 40, 50]]), array([[300, 400, 500]])]]

Upvotes: 1

rapchen
rapchen

Reputation: 31

Maybe you can try zip function:

trans_format = list(zip(*all_l))

But then each element of the outer list will be a tuple. If you need inner lists you can simply add a conversion:

trans_format = [list(row) for row in zip(*all_l)]

Upvotes: 0

Related Questions