CODEWITHSUNDEEP
numpynan
Mad Physicist
Mad Physicist

Reputation: 114330

Setting a value in masked location with NaNs present in numpy

I have an array with NaNs, say

>>> a = np.random.randn(3, 3)
>>> a[1, 1] = a[2, 2] = np.nan
>>> a
array([[-1.68425874,  0.65435007,  0.55068277],
       [ 0.71726307,         nan, -0.09614409],
       [-1.45679335, -0.12772348,         nan]])

I would like to set negative numbers in this array to -1. Doing this the "straightforward" way results in a warning, which I am trying to avoid:

>>> a[a < 0] = -1
__main__:1: RuntimeWarning: invalid value encountered in less
>>> a
array([[-1.        ,  0.65435007,  0.55068277],
       [ 0.71726307,         nan, -1.        ],
       [-1.        , -1.        ,         nan]])

Applying AND to the masks results in the same warning because of course a < 0 is computed as a separate temp array:

>>> n = ~np.isnan(a)
>>> a[n & (a < 0)] = -1
__main__:1: RuntimeWarning: invalid value encountered in less

When I try to apply a mask the nans out of a, the masked portion is not written back to the original array:

>>> n = ~np.isnan(a)
>>> a[n][a[n] < 0] = -1
>>> a
array([[-1.68425874,  0.65435007,  0.55068277],
       [ 0.71726307,         nan, -0.09614409],
       [-1.45679335, -0.12772348,         nan]])

The only way I could figure out of solving this is by using a gratuitous intermediate masked version of a:

>>> n = ~np.isnan(a)
>>> b = a[n]
>>> b[b < 0] = -1
>>> a[n] = b
>>> a
array([[-1.        ,  0.65435007,  0.55068277],
       [ 0.71726307,         nan, -1.        ],
       [-1.        , -1.        ,         nan]])

Is there a simpler way to perform this masked assignment with the presence of NaNs? I would like to solve this without the use of masked arrays if possible.

NOTE

The snippets above are best run with

import numpy as np
import warnings
np.seterr(all='warn')
warnings.simplefilter("always")

as per https://stackoverflow.com/a/30496556/2988730.

Upvotes: 1

Views: 723

Answers (3)

hpaulj
hpaulj

Reputation: 231385

Poking around the np.nan... functions I found np.nan_to_num

In [569]: a=np.arange(9.).reshape(3,3)-5
In [570]: a[[1,2],[1,2]]=np.nan
In [571]: a
Out[571]: 
array([[ -5.,  -4.,  -3.],
       [ -2.,  nan,   0.],
       [  1.,   2.,  nan]])
In [572]: np.nan_to_num(a)   # replace nan with 0
Out[572]: 
array([[-5., -4., -3.],
       [-2.,  0.,  0.],
       [ 1.,  2.,  0.]])
In [573]: np.nan_to_num(a)<0    # and safely do the <
Out[573]: 
array([[ True,  True,  True],
       [ True, False, False],
       [False, False, False]], dtype=bool)
In [574]: a[np.nan_to_num(a)<0]=-1
In [575]: a
Out[575]: 
array([[ -1.,  -1.,  -1.],
       [ -1.,  nan,   0.],
       [  1.,   2.,  nan]])

Looking at the nan_to_num code, it looks like it uses a masked copyto:

In [577]: a1=a.copy(); np.copyto(a1, 0.0, where=np.isnan(a1))
In [578]: a1
Out[578]: 
array([[-1., -1., -1.],
       [-1.,  0.,  0.],
       [ 1.,  2.,  0.]])

So it's like your version with the 'gratuitous' mask, but it's hidden in the function.

np.place, np.putmask are other functions that use a mask.

Upvotes: 1

ev-br
ev-br

Reputation: 26050

You can suppress the warning temporarily, is this what you're after?

In [9]: a = np.random.randn(3, 3)

In [10]: a[1, 1] = a[2, 2] = np.nan

In [11]: with np.errstate(invalid='ignore'):
   ....:     a[a < 0] = -1
   ....:

Upvotes: 2

Divakar
Divakar

Reputation: 221574

If you want to avoid that warning occurring at a < 0 with a containing NaNs, I would think alternative ways would involve using flattened or row-column indices of non-Nan positions and then performing the comparison. Thus, we would have two approaches with that philosophy.

One with flattened indices -

idx = np.flatnonzero(~np.isnan(a))
a.ravel()[idx[a.ravel()[idx] < 0]] = -1

Another with subscripted-indices -

r,c = np.nonzero(~np.isnan(a))
mask = a[r,c] < 0
a[r[mask],c[mask]] = -1

Upvotes: 2

Related Questions