CODEWITHSUNDEEP
pythonnumpyscipy
user3410943
user3410943

Reputation:

median-absolute-deviation (MAD) based outlier detection

I wanted to apply median-absolute-deviation (MAD) based outlier detection using the answer from @Joe Kington as given below:

Pythonic way of detecting outliers in one dimensional observation data

However, what's going wrong with my code, I could not figure out how to assign the outliers as nan values for MY DATA:

import numpy as np
data = np.array([55,32,4,5,6,7,8,9,11,0,2,1,3,4,5,6,7,8,25,25,25,25,10,11,12,25,26,27,28],dtype=float)
median = np.median(data, axis=0)
diff = np.sum((data - median)**2, axis=-1)
diff = np.sqrt(diff)
med_abs_deviation = np.median(diff)
modified_z_score = 0.6745 * diff / med_abs_deviation
data_without_outliers = data[modified_z_score < 3.5]
?????
print data_without_outliers

Upvotes: 6

Views: 13375

Answers (2)

tstanisl
tstanisl

Reputation: 14127

The problem might be line:

diff = np.sum((data - median)**2, axis=-1)

Applying np.sum() will collapse the result to scalar. Remove top-level sum, and your code will work. Other way around it to ensure that that data is at least 2d array. You can use numpy.atleast_2d() for that.

In order to assign NaNs, follow answer from https://stackoverflow.com/a/22804327/4989451

Upvotes: 1

ebarr
ebarr

Reputation: 7842

What is the problem with using:

data[modified_z_score > 3.5] = np.nan

Note that this will only work if data is a floating point array (which it should be if you are calculating MAD).

Upvotes: 1

Related Questions