CODEWITHSUNDEEP
pythonpandasnull
WalksB
WalksB

Reputation: 519

How to check if Pandas/NumPy arbitrary object contains or IS NaT/NaN/Null

I want to check if a Pandas object is/contains any Null/NaN/NaT value, but I have no information beforehand if the object is a list or one singular value.

I tried

x = [1,2,3,pd.NaT]
if pd.notnull(x):
    ...

But if the object x is a list, it returns this value error (due to it returning an array of boolean values):

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

And if I do this:

x = pd.NaT
if pd.notnull(x).any():
   ...

It returns this error if I receive a singular value:

AttributeError: 'bool' object has no attribute 'any'

What's the cleanest way to go about this that is capable of dealing with both lists that may contain NaN's AND NaN's themselves?

Upvotes: 1

Views: 694

Answers (2)

Andy L.
Andy L.

Reputation: 25269

Wrap x inside of list and pass to pd.notna and chain any. It works because pd.notna returns numpy ndarray. Therefore, the any is actually ndarray.any. The When calling numpy ndarray.any with out axis parameter, it will check on all dimensions. Therefore, it works on both list x or single value x

x = [1,2,3,pd.NaT]

In [369]: pd.notna([x])
Out[369]: array([[ True,  True,  True, False]]) #it is 2d-array

In [370]: type(pd.notna([x]))
Out[370]: numpy.ndarray

In [373]: pd.notna([x]).any()  #`ndarray.any` checks on all dimensions of this 2d-array
Out[373]: True

In [374]: pd.notna([x]).all()  #`ndarray.all` checks on all dimensions of this 2d-array
Out[374]: False

On x is single pd.NaT

x = pd.NaT

In [377]: pd.notna([x])
Out[377]: array([False])  #it is 1d-array

In [378]: pd.notna([x]).any()
Out[378]: False

In [379]: pd.notna([x]).all()
Out[379]: False

Upvotes: 1

MARCO LAGALLA
MARCO LAGALLA

Reputation: 267

If you only need to know if any value in your dataframe is a NaN, then you can simply do

if df.isnull().any().any():
    # NaN present(s)
    do_something()

Upvotes: 0

Related Questions