Invalid value encountered in greater - numpy array logic

Question

The question "Why do I get an invalid value ..." has been discussed a few times already. Here is another variant which I don't quite understand yet - this is why I am opening this thread:

(1) in

y = np.nan
(~np.isnan(y)) & (y > 5.)

I get the correct result and no error is thrown.

(2) However, in

y = np.array([np.isnan, 6.])
(~np.isnan(y)) & (y > 5.)

the "Invalid value encountered in greater" warning is shown.

This means that the short-circuiting (Python reference) in the boolean and expression does not work if this expression is used with numpy arrays.

Digging a bit on stackoverflow (e.g. ref ) and elsewhere it seems that under the hood the array expression is written as

np.logical_and.reduce([~np.isnan(y), y>5.])

This means that both expressions are first evaluated for all elements before they are combined (indeed, this produces the same warning). Could someone please confirm that this is what is happening? And does anyone have asolution for this problem other than looping over all array elements?

The context I am using this expression in is to set additional values in an array to nan if they exceed a threshold, i.e. the actual code has something like

y[(~np.isnan(y)) & (y > 5.)] = np.nan

Answer 1

This can do it, with some boolean indexing to avoid the nan and a where statement

y[np.isfinite(y)] = np.where(y[np.isfinite(y)] > 5, np.nan, y[np.isfinite(y)])

Answer 2

OK: seems I have found a solution myself:

array_and = np.frompyfunc(lambda x: (~np.isnan(x)) & (x > 5.), 1, 1)
array_and(y)

appears to work. Reference: [1]