CODEWITHSUNDEEP
pythonnumpymultidimensional-arrayfeature-engineering
ppiont
ppiont

Reputation: 13

Shifting values with np.apply_along_axis returns inconsistent results

TL;DR: np.apply_along_axis works for a certain array with shape (1561, 338) which is a subset of another array with shape (351225, 338) for which it fails.

I am trying to apply the following function:

def add_min(a):
    return a + abs(a.min()) if a.min() < 0 else a

x_train has shape (1561, 15, 15, 338) (n * height * width * channels) and I need to shift all values to positive to be able to log normalize my data. I want to do that per channel, for obvious reasons.

Now if I reshape x_train: x_train = x_train.reshape(-1, 338) and get shape (351225, 338) I should be able to perform:

x_train = np.apply_along_axis(add_min, 0, x_train)

However...

Before:

x_train.min()
>> -2147483648

After:

x_train.min()
>> -2147370103

In other words, it does not work. On the other hand, if I only keep the center pixel:

# Keep the center value of axes (1, 2)
x_train = x_train[:, x_train.shape[1]//2, x_train.shape[2]//2, :]
x_train.shape
>> (1561, 338)

x_train.min()
>> -32768  # strange coincidence that this location in the image has a much smaller value range 

x_train = np.apply_along_axis(add_min, 0, x_train)
x_train.min()
>> 0

I think it has something to do with the large negative values, because if I select random indices in the 2 center axes (i.e. 1 and 8) instead of the middle (7, 7) I again get x_train.min() of -2147483648 and -2147369934, before and after np.apply_along_axis, respectively.

So what am I doing wrong? Is there a better way I can achieve my goal?

Upvotes: 0

Views: 51

Answers (1)

hpaulj
hpaulj

Reputation: 231615

The overflow on int32 is a good guess if the problem is "random". But using apply_along_axis may be making it harder to diagnose the issue, since it's wrapping your function in an (obscure) iteration. It should be easier to diagnose things with whole array calculations.

Make a modest array a mix of min values:

In [77]: A = np.random.randint(-50,1000,(4,8))
In [78]: A
Out[78]: 
array([[151, 531, 765, 379,  89, 499, 818, 848],
       [873, -12, -45, 900, 416, 838, 603, 849],
       [540,   0,   1, 589, 297, 566, 688, 556],
       [ 53, 170, 461, -16,  -6, 480, 321, 392]])

Your function:

In [79]: np.apply_along_axis(add_min, 0, A)
Out[79]: 
array([[151, 543, 810, 395,  95, 499, 818, 848],
       [873,   0,   0, 916, 422, 838, 603, 849],
       [540,  12,  46, 605, 303, 566, 688, 556],
       [ 53, 182, 506,   0,   0, 480, 321, 392]])

Let's create an whole-array equivalent. First find the min with a axis specification:

In [80]: am = np.min(A, axis=0, keepdims=True)
In [81]: am
Out[81]: array([[ 53, -12, -45, -16,  -6, 480, 321, 392]])

Now create a shift array that imitates your function (without the if that only works for scalars):

In [82]: shift=np.abs(am)
In [83]: shift[am>=0]=0
In [84]: shift
Out[84]: array([[ 0, 12, 45, 16,  6,  0,  0,  0]])
In [85]: A+shift
Out[85]: 
array([[151, 543, 810, 395,  95, 499, 818, 848],
       [873,   0,   0, 916, 422, 838, 603, 849],
       [540,  12,  46, 605, 303, 566, 688, 556],
       [ 53, 182, 506,   0,   0, 480, 321, 392]])

There are other ways of getting that shift, but the same basic idea applies, using the am<0 to determine which columns get the shift.

This will also be faster.

Upvotes: 0

Related Questions