CODEWITHSUNDEEP
numpy
Hello12345
Hello12345

Reputation: 21

Fill nan in numpy array

Is there a straight forward way of filling nan values in a numpy array when the left and right non nan values match?

For example, if I have an array that has False, False , NaN, NaN, False, I want the NaN values to also be False. If the left and right values do not match, I want it to keep the NaN

Upvotes: 1

Views: 706

Answers (2)

DomTomCat
DomTomCat

Reputation: 8569

See also Most efficient way to forward-fill NaN values in numpy array

Pandas offers the interpolate method on data frames. You could use it like this:

interpolate(limit_direction='both').to_numpy().flatten()

A much more specialized method is given below, it allows controlling the B-Spline interpolation parameters, extrapolation and smoothing:

https://github.com/DomTomCat/signalProcessing/blob/main/interpolateNonFiniteValues.py

Upvotes: 0

hpaulj
hpaulj

Reputation: 231385

Your first task is to reliably identify the np.nan elements. Because it's a unique float value, testing isn't trivail. np.isnan is the best numpy tool.

To mix boolean and float (np.nan) you have to use object dtype:

In [68]: arr = np.array([False, False, np.nan, np.nan, False],object)                                        
In [69]: arr                                                                                                 
Out[69]: array([False, False, nan, nan, False], dtype=object)

converting to float changes the False to 0 (and True to 1):

In [70]: arr.astype(float)                                                                                   
Out[70]: array([ 0.,  0., nan, nan,  0.])

np.isnan is a good test for nan, but it only works on floats:

In [71]: np.isnan(arr)                                                                                       
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-71-25d2f1dae78d> in <module>
----> 1 np.isnan(arr)

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

In [72]: np.isnan(arr.astype(float))                                                                         
Out[72]: array([False, False,  True,  True, False])

You could test the object array (or a list) with a helper function and a list comprehension:

In [73]: def foo(x): 
    ...:     try: 
    ...:         return np.isnan(x) 
    ...:     except TypeError: 
    ...:         return x 
    ...:                                                                                                     
In [74]: [foo(x) for x in arr]                                                                               
Out[74]: [False, False, True, True, False]

Having reliably identified the nan values, you can then apply the before/after logic. I'm not sure if it's easier with lists or array (your logic isn't entirely clear).

Upvotes: 2

Related Questions