CODEWITHSUNDEEP
pythonpandasnumpymissing-data
willk
willk

Reputation: 3817

Numpy inconsistent results with Pandas and missing values

Why does numpy return different results with missing values when using a Pandas series compared to accessing the series' values as in the following:

import pandas as pd
import numpy as np

data = pd.DataFrame(dict(a=[1, 2, 3, np.nan, np.nan, 6]))
np.sum(data['a'])

#12.0

np.sum(data['a'].values)
#nan

Upvotes: 5

Views: 342

Answers (1)

cs95
cs95

Reputation: 402413

Calling np.sum on a pandas Series delegates to Series.sum, which ignores NaNs when computing the sum (BY DEFAULT).

data['a'].sum()
# 12.0

np.sum(data['a'])
# 12.0

You can see this from the source code of np.sum:

np.sum??

def sum(a, axis=None, dtype=None, out=None, keepdims=np._NoValue, initial=np._NoValue):
    ...
    return _wrapreduction(a, np.add, 'sum', axis, dtype, out, keepdims=keepdims,

Taking a look at the source code for _wrapreduction, we see:

np.core.fromnumeric._wrapreduction??

def _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs):
    ...
    if type(obj) is not mu.ndarray:
        try:
            reduction = getattr(obj, method)   # get reference to Series.add

reduction is then finally called at the end of the function:

            return reduction(axis=axis, out=out, **passkwargs)

Upvotes: 5

Related Questions