Building and shifting numpy array

Question

I have such a numpy array:

And I want to shift this array with N so I can get a new array, such that first 4 rows will be the first 4 rows from the beginning array, and the below rows will be the rows from the previous step.

I dont know how to explain it correctly, but look at the example where I shift with 1 (not the whole array, but I would like to do this for the whole array):

And when I shift with 2 (not the whole array, but I would like to do this for the whole array):

And if I shift with 3, I will have 12 zeros in first column, and 3 previous rows ..

And here's the code:

import numpy as np
import numpy 

def shift_one(arr):
  arrT = arr.transpose()
  arrT_shape = arrT.shape[0]

  col1 = arrT[:, [0]]
  prev1 = numpy.zeros([arrT_shape, 1])   
  x1 = numpy.vstack([col1, prev1])

  col2 = arrT[:, [1]]
  prev2 = arrT[:, [0]]
  x2 = numpy.vstack([col2, prev2])

  col3 = arrT[:, [2]]
  prev3 = arrT[:, [1]]
  x3 = numpy.vstack([col3, prev3])

  col4 = arrT[:, [3]]
  prev4 = arrT[:, [2]]
  x4 = numpy.vstack([col4, prev4])

  col5 = arrT[:, [4]]
  prev5 = arrT[:, [3]]
  x5 = numpy.vstack([col5, prev5])

  # ... and so on, until we have index [23] and [22] and x24

  res = numpy.hstack([x1, x2, x3, x4, x5])
  return res 

def shift_two(arr):
    arrT = arr.transpose()
    arrT_shape = arrT.shape[0]
    new_size = 2 * arrT_shape

    col1 = arrT[:, [0]]
    prev1 = numpy.zeros([new_size, 1])
    x1 = numpy.vstack([col1, prev1])

    col22 = arrT[:, [1]]
    col21 = arrT[:, [0]]
    prev2 = numpy.zeros([arrT_shape, 1])
    x2 = numpy.vstack([col22, col21, prev2])

    col32 = arrT[:, [2]]
    col31 = arrT[:, [1]]
    col30 = arrT[:, [0]]
    x3 = numpy.vstack([col32, col31, col30])

    col42 = arrT[:, [3]]
    col41 = arrT[:, [2]]
    col40 = arrT[:, [1]]
    x4 = numpy.vstack([col42, col41, col40])

    col52 = arrT[:, [4]]
    col51 = arrT[:, [3]]
    col50 = arrT[:, [2]]
    x5 = numpy.vstack([col52, col51, col50])

    # ... and so on, until we have index [23], [22], [21] and x24

    res = numpy.hstack([x1, x2, x3, x4, x5])
    return res

arr1 = np.array([[0,   2,   0, 324], 
        [1, 2, 0,324], 
        [2, 2, 0, 324], 
        [3, 2, 0, 324], 
        [4, 2, 0, 324], 
        [5, 2, 0, 324], 
        [6, 2, 0, 324], 
        [7, 2, 0, 324], 
        [8, 2, 0, 324], 
        [9, 2, 0, 324], 
        [ 10, 2, 0, 324], 
        [ 11, 2, 0, 324], 
        [ 12, 2, 0, 324], 
        [ 13, 2, 0, 324], 
        [ 14, 2, 0, 324], 
        [ 15, 2, 0, 324], 
        [ 16, 2, 0, 324], 
        [ 17, 2, 0, 324], 
        [ 18, 2, 0, 324], 
        [ 19, 2, 0, 324], 
        [ 20, 2, 0, 324], 
        [ 21, 2, 0, 324], 
        [ 22, 2, 0, 324], 
        [ 23, 2, 0, 324]])

print(arr1)

print('\n\n')
one = shift_one(arr1)
print(one)

print('\n\n')
two = shift_two(arr1)
print(two)

Basically, I have a problem how to write a function, which will shift it with a given N ... I know how to write it step by step as I did, but I have a problem transforming it into something more usable. Thank you.

So again, an example for array with 5 columns (the original array has 24 columns):

Answer 1

Looped

If you want to do this for the general case, you'll can start with a simple loop. If the input array has shape (M, N), the output will be (M * (delay + 1), N).

Stacking is one way to combine arrays. You can minimize the required number of stacks by always doubling the result:

def stagger(a, delay):
    m = a.shape[0]
    z = 1
    d = delay + 1
    while z < d:
        k = min(d - z, z)    # This will only kick in for the last iteration
        new = np.concatenate([np.zeros((m * k, z)), a[:m * k, :-z]], axis=1)
        a = np.concatenate([a, new], axis=0)
        z *= 2
    return a

Vectorized

That being said, there is a fully vectorized way to do this using the monster that is np.lib.stride_tricks.asstrided. Imagine an array that is the first row of your original data padded on the left with delay + 1 zeros:

>>> d = delay + 1    # delay = 7
>>> row = np.concatenate([np.zeros(d), a[0]])
>>> row
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7., 
        8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.])

You can view this array (without copying the data) as a staggered 2D buffer by adding a fake second dimension whose stride is one element shorter than it ought to be:

>>> m, n = a.shape
>>> staggered_row = np.lib.stride_tricks.as_strided(row[d:], shape=(d, n), strides=(-row.strides[0], row.strides[0]))
>>> staggered_row
array([[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11.,
        12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.],
       [ 0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.,
        11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.],
       [ 0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.,
        10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.],
       [ 0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,
         9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
       [ 0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,
         8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,
         7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,
         6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,
         5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16.]])

Here, the negative stride adds the elements of the zero padding into the array. Keep in mind that this is generally not safe and should be done with extreme care. The output will not be writable by default, because all the rows refer to largely the same memory locations.

Now imagine doing the same thing to your entire array: padding and adding a new dimension that lets you step back into the padding. If you order the dimensions carefully, a reshape of the view will create a new copy that looks exactly how you want in one step. Specifically, starting with shape (M, N), we pad to (D + M, N), create a view that is (D, M, N), and reshape to (D * M, N):

def staggerx(a, delay):
    d = delay + 1
    m, n = a.shape
    padded = np.concatenate([np.zeros((m, d)), a], axis=1)
    view = np.lib.stride_tricks.as_strided(padded[:, d:], shape=(d, m, n), strides=(-padded.strides[1], padded.strides[0], padded.strides[1]))
    result = view.reshape(-1, n)
    return result

A reshape of a crazily strided array will always make a copy because the data is not contiguous by the way.

Answer 2

If the data is not too humongous concatenating a couple of times will work:

>>> x
array([[  0,   1,   2,   3,   4],
       [  2,   2,   2,   2,   2],
       [  0,   0,   0,   0,   0],
       [324, 324, 324, 324, 324]])
>>> np.concatenate([x] + [np.concatenate([np.zeros(x.shape[0]*i).reshape(-1, i), x[: , i:]] ,axis=1) for i in range(1,x.shape[1])], axis=0)
array([[  0.,   1.,   2.,   3.,   4.],
       [  2.,   2.,   2.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [324., 324., 324., 324., 324.],
       [  0.,   1.,   2.,   3.,   4.],
       [  0.,   2.,   2.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0., 324., 324., 324., 324.],
       [  0.,   0.,   2.,   3.,   4.],
       [  0.,   0.,   2.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0.,   0., 324., 324., 324.],
       [  0.,   0.,   0.,   3.,   4.],
       [  0.,   0.,   0.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0., 324., 324.],
       [  0.,   0.,   0.,   0.,   4.],
       [  0.,   0.,   0.,   0.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0., 324.]])

Each time I generate a 4xi matrix of zeros, and take the i+1: columns from the original matrix to generate the next part. Finally, I concatenate all these arrays.