Are there rules for the interaction between numpy reshape() and transpose()?

Question

I've put this question in quite a bit of context, to hopefully make it easier to understand, but feel free to skip down to the actual question.

Context

Here is the work I was doing which sparked this question:

I'm working with an API to access some tabular data, which is effectively a labelled N-dimensional array. The data is returned as a flattened list of lists (of the actual data values), plus a list of the different axes and their labels, e.g.:

raw_data = [
    ['nrm', 'nrf'],
    ['ngm', 'ngf'],
    ['nbm', 'nbf'],
    ['srm', 'srf'],
    ['sgm', 'sgf'],
    ['sbm', 'sbf'],
    ['erm', 'erf'],
    ['egm', 'egf'],
    ['ebm', 'ebf'],
    ['wrm', 'wrf'],
    ['wgm', 'wgf'],
    ['wbm', 'wbf'],
]

axes = [
    ('Gender', ['Male', 'Female']),
    ('Color', ['Red', 'Green', 'Blue']),
    ('Location', ['North', 'South', 'East', 'West']),
]

The data is normally numeric, but I've used strings here so you can easily see how it matches up with the labels, e.g. nrm is the value for North, Red, Male.

The data loops through axis 0 as you go across (within) a list, and then loops through axes 1 and 2 as you go down the lists, with axis 1 (on the "inside") varying most rapidly, then 2 (and for higher-dimensional data continuing to work "outwards"), viz:

       axis 0 ->
a a [ # # # # # # ]
x x [ # # # # # # ]
i i [ # # # # # # ]
s s [ #  R A W  # ]
    [ # D A T A # ]
2 1 [ # # # # # # ]
↓ ↓ [ # # # # # # ]
    [ # # # # # # ]

I want to reshape this data and match it up with its labels, which I did using the following to output it into a Pandas (multi-index) DataFrame:

import numpy as np
import pandas as pd

names = [name for (name, _) in axes]
labels = [labels for (_, labels) in axes]

sizes = tuple(len(L) for L in labels)  # (2, 3, 4)
data_as_array = np.array(raw_data)  # shape = (12, 2) = (3*4, 2)
A = len(sizes)  # number of axes
new_shape = (*sizes[1:],sizes[0])  # (3, 4, 2)

data = data_as_array.reshape(new_shape, order="F").transpose(A - 1, *range(A - 1))
# With my numbers: data_as_array.reshape((3, 4, 2), order="F").transpose(2, 0, 1)

df = pd.DataFrame(
    data.ravel(),
    index=pd.MultiIndex.from_product(labels, names=names),
    columns=["Value"],
)

(I've noted with comments what some of the particular values are for my example, but the code is meant to be generalised for any N-dimensional data.)

This gives:

                      Value
Gender Color Location      
Male   Red   North      nrm
             South      srm
             East       erm
             West       wrm
       Green North      ngm
             South      sgm
             East       egm
             West       wgm
       Blue  North      nbm
             South      sbm
             East       ebm
             West       wbm
Female Red   North      nrf
             South      srf
             East       erf
             West       wrf
       Green North      ngf
             South      sgf
             East       egf
             West       wgf
       Blue  North      nbf
             South      sbf
             East       ebf
             West       wbf

This is all as desired and expected, and you can see that the values have ended up in the correct places, i.e. attached to their matching labels.

Question

My actual question concerns this line:

data = data_as_array.reshape(new_shape, order="F").transpose(A - 1, *range(A - 1))

which with the specific numbers in my example was:

data = data_as_array.reshape((3, 4, 2), order="F").transpose(2, 0, 1)

After some experimentation, I discovered that all three of the following are equivalent (the first is the original version):

data1 = data_as_array.reshape(new_shape, order="F").transpose(D - 1, *range(D - 1))
data2 = data_as_array.T.reshape(*reversed(new_shape)).T.transpose(D - 1, *range(D - 1))
data3 = data_as_array.reshape(*reversed(sizes)).T

But this got me thinking (and here is my question at last!):

Are there any rules that I could use to manipulate the expression, to get from e.g. data1 to data3?

In particular, it seems like transpose() and reshape() are closely linked and that there might be a way to "absorb" the action of the tranpose into the reshape(), so that you can drop it or at least transform it into a neater .T (as per data3).

My attempt

I managed to establish the following rule:

a.reshape(shape, order="F") == a.T.reshape(*reversed(shape)).T

You can apply .T to both sides, or substitute a.T in for a to get these variations of it:

a.reshape(shape) == a.T.reshape(*reversed(shape), order="F").T
a.reshape(shape).T == a.T.reshape(*reversed(shape), order="F")
a.T.reshape(shape) == a.reshape(*reversed(shape), order="F").T

a.reshape(shape, order="F") == a.T.reshape(*reversed(shape)).T
a.reshape(shape, order="F").T == a.T.reshape(*reversed(shape))
a.T.reshape(shape, order="F") == a.reshape(*reversed(shape)).T

I think this is effectively the definition of the difference between row-major and column-major ordering, and how they relate.

But what I haven't managed to do is show is how you can go from:

data = data_as_array.reshape((3, 4, 2), order="F").transpose(2, 0, 1)

to:

data = data_as_array.reshape((4, 3, 2))

So somehow put the transposition into the reshape.

But I'm not even sure if this is generally true, or is specific to my data or e.g. 3 dimensions.

EDIT: To clarify, I'm reasonably happy with how a straight-up .T transpose works, and the rules above cover that. (Note that .T is equivalent to .tranpose(2, 1, 0) for 3 axes, or .tranpose(n-1, n-2, ... 2, 1, 0) for the general case of n axes.)

It's the case of using .transpose() where you're doing a "partial" transpose that I'm curious about, e.g. .tranpose(1, 0, 2) - where you're doing something other than just reversing the order of the axes.

Some references:

This covers row-major and column-major differences: How do you unroll a Numpy array of (mxn) dimentions into a single vector (and I can quite easily see how that's happening in my data)
This SO answer is really helpful in explaining transposing, and basically covers reshaping as well: https://stackoverflow.com/a/32034565/9219425 (check out the fantastic diagrams!), including covering how transposing affects shape and strides. I wrote an algorithm mimicking this process to see if that would make things clearer (e.g. transposing might correspond to swapping the order of the for loops in the algorithm), but it didn't really help.

Are there rules for the interaction between numpy reshape() and transpose()?

Context

Question

My attempt

Answers (1)

Related Questions