CODEWITHSUNDEEP
pythonpandasreplacenanoutliers
Sridhar
Sridhar

Reputation: 121

Pandas: replacing outliers (3 sigma) in all numerical columns of a dataframe with NaN

I have a data frame with numerical and string columns. 

import numpy as np 
import pandas as pd 
from scipy.stats import zscore

data = {'c1' : [1., 2., 3., 4.], 'c2' : [4., 3., 2., 1.], 'c3' : [5., 6., 7000., 8.], 
        'c4' : [8., 7., 6., 10000.], 'c5' : ['a', 'b', 'c', 'd']}

I want to replace the outliers in numerical columns with NaN.

    c1  c2  c3  c4  c5  
0   1.0 4.0 5.0 8.0 a  
1   2.0 3.0 6.0 7.0 b  
2   3.0 2.0 NaN 6.0 c  
3   4.0 1.0 8.0 NaN d

This code does what I want to do. 

df = pd.DataFrame(data) 
allcol = list(df) 
numcol = [x for x in allcol if x not in ('c5')] 
df[numcol] = df[numcol].mask(~df[numcol].apply(lambda x: zscore(x) < 1.5, axis=1))

Wondering if you know any better and simpler solution...

Upvotes: 1

Views: 1915

Answers (1)

Scott Boston
Scott Boston

Reputation: 153460

You could set 'c5' into the index and then use:

df1 = df.set_index('c5')
df1.where(df1.apply(zscore).lt(1.5)).reset_index().reindex_axis(df.columns,1)

Output:

    c1   c2   c3   c4 c5
0  1.0  4.0  5.0  8.0  a
1  2.0  3.0  6.0  7.0  b
2  3.0  2.0  NaN  6.0  c
3  4.0  1.0  8.0  NaN  d

Upvotes: 2

Related Questions