Reshape of set of arrays in 3D array

Question

In order to do calculations, I have a set of arrays: "sub" array (as you can see below), and I want to reshape it in an array as given by "test" array:

import numpy as np

sub = np.array([[[[ 1.,  1.],
         [ 1.,  1.]],

        [[ 2.,  2.],
         [ 2.,  2.]],

        [[ 3.,  3.],
         [ 3.,  3.]],

        [[ 4.,  4.],
         [ 4.,  4.]]],

       [[[ 5.,  5.],
         [ 5.,  5.]],

        [[ 6.,  6.],
         [ 6.,  6.]],

        [[ 7.,  7.],
         [ 7.,  7.]],

        [[ 8.,  8.],
         [ 8.,  8.]]]])


test=np.array([[[ 1.,  1.,  2., 2.],
        [ 1.,  1.,  2., 2.],
        [ 3.,  3.,  4., 4.],
        [ 3.,  3.,  4., 4.]],

       [[ 5.,  5.,  6., 6.],
        [ 5.,  5.,  6., 6.],
        [ 7.,  7.,  8., 8.],
        [ 7.,  7.,  8., 8.]]]) 

I have found on a post a part of code which seems to work for my case, but I have some errors...

k,l,m,n,p =2,2,2,2,2
conc = np.array([np.ones([p,m,n],dtype=int)*i for i in range(k*l)])
test_reshape=np.vstack([np.hstack(sub[i:i+l]) for i in range(0,k*l*p,l)])

Answer 1

This can be done using a reshape/swapaxes trick:

In [92]: sub.reshape(2,2,2,2,2).swapaxes(2,3).reshape(test.shape)
Out[92]: 
array([[[ 1.,  1.,  2.,  2.],
        [ 1.,  1.,  2.,  2.],
        [ 3.,  3.,  4.,  4.],
        [ 3.,  3.,  4.,  4.]],

       [[ 5.,  5.,  6.,  6.],
        [ 5.,  5.,  6.,  6.],
        [ 7.,  7.,  8.,  8.],
        [ 7.,  7.,  8.,  8.]]])

In [94]: np.allclose(sub.reshape(2,2,2,2,2).swapaxes(2,3).reshape(test.shape), test)
Out[94]: True

I confess I do not know how to generate this kind of solution without some guessing. But it appears that when you want to rearrange "blocks" in an array, there is a way to do it by reshaping to a higher dimension, swapping some axes, then reshaping to the desired shape. Given that sub.shape is (2, 4, 2, 2) reshaping to a higher dimension must mean (2, 2, 2, 2, 2). So you only have to test for a solution of the form

sub.reshape(2,2,2,2,2).swapaxes(i,j).reshape(test.shape)

and that is easy to do:

for i,j in IT.combinations(range(5), 2):
    if np.allclose(sub.reshape(2,2,2,2,2).swapaxes(i,j).reshape(test.shape), test):
       print(i,j)

reveals the right axes to swap:

(2, 3)

Answer 2

import numpy as np
sub = np.array(...)
test = np.array([np.hstack((np.vstack(( s[0],s[1] )),
                            np.vstack(( s[2],s[3] )))) for s in sub])
print test

In the OP's example the shape of sub is (2,4,2,2), but that the code above would work as is for an array of shape (n,4,m,m). For different shapes of the type (n,k,m,m) the code above can be adapted to different requirements.

Eventually I would like to add that when you look at the code you literally see what the code is achieving, and this may be compensating other defects of the code in terms of efficiency (i.e., copying vs reshaping).

---

A better solution (i.e, not mine ;-) and some aftertoughts

I have found this answer from unutbu (that contains a link to a more general solution) that the OP can easily (?) adapt to her/his needs. Due to the complex reshaping that is involved data is however copied, hence the OP may want to measure the different performances of the two approaches, taking into account the incidence of the "reshaping" on the total run time of her/his program (i.e., imho shaving 0.3s on a runtime of 2' wouldn't be worth the effort)

Examplar interactive session

In the following, the data and the procedures are literally lifted from the above mentioned answer from unutbu, with the last two statements added by me to show the addresses of the data buffers of the three ndarrays, x, y and z.

In [1]: import numpy as np

In [2]: x = np.arange(16).reshape((4,2,2))

In [3]: y = x.reshape(2,2,2,2).swapaxes(1,2).reshape(4,-1)

In [4]: x
Out[4]: 
array([[[ 0,  1],
        [ 2,  3]],

       [[ 4,  5],
        [ 6,  7]],

       [[ 8,  9],
        [10, 11]],

       [[12, 13],
        [14, 15]]])

In [5]: y
Out[5]: 
array([[ 0,  1,  4,  5],
       [ 2,  3,  6,  7],
       [ 8,  9, 12, 13],
       [10, 11, 14, 15]])

In [6]: z = x.T

In [7]: [a.__array_interface__['data'][0] for a in (x, y, z)]
Out[7]: [46375856, 45578800, 46375856]

In [8]: 

Answer 3

Here's an alternative way to swap, slice and stack your array into shape:

>>> t = sub.swapaxes(1, 3).T.swapaxes(1, 3)
>>> x = np.c_[t[::2, 0], t[1::2, 0]]
>>> y = np.c_[t[::2, 1], t[1::2, 1]]
>>> np.array((np.r_[x[0], x[1]], np.r_[y[0], y[1]]))

array([[[ 1.,  1.,  2.,  2.],
        [ 1.,  1.,  2.,  2.],
        [ 3.,  3.,  4.,  4.],
        [ 3.,  3.,  4.,  4.]],

       [[ 5.,  5.,  6.,  6.],
        [ 5.,  5.,  6.,  6.],
        [ 7.,  7.,  8.,  8.],
        [ 7.,  7.,  8.,  8.]]])

Edit: Or instead, squeeze, slice and stack:

>>> x = np.c_[sub[:1][:,::2], sub[:1][:,1::2]].squeeze()
>>> y = np.c_[sub[1:][:,::2], sub[1:][:,1::2]].squeeze()
>>> np.array((np.r_[x[0], x[1]], np.r_[y[0], y[1]]))
# the required array

Answer 4

Perhaps there exists a pure numpy solution, but I'm not aware of it and it'll use quite a few tricks with strides. The solution below is thus not as efficient, because it uses python's for loops (making it less quick), but it 'll get your result in a general way, so without it depending on the size of your actual 4D array.

np.vstack( (sub[vol,2*sheet:2*sheet+2].reshape((4,-1)).T for vol in range(2) for sheet in range(2))).reshape((2,4,-1)