CODEWITHSUNDEEP
python-3.xpandasdataframemissing-datadrop
Giampaolo Levorato
Giampaolo Levorato

Reputation: 1622

How to drop entire record if more than 90% of features have missing value in pandas

I have a pandas dataframe called df with 500 columns and 2 million records.

I am able to drop columns that contain more than 90% of missing values.

But how can I drop in pandas the entire record if 90% or more of the columns have missing values across the whole record?

I have seen a similar post for "R" but I am coding in python at the moment.

Upvotes: 0

Views: 2430

Answers (2)

Niels Henkens
Niels Henkens

Reputation: 2696

You can use df.dropna() and set the thresh parameter to the value that corresponds to 10% of your columns (the minimum number of non-NA values).

df.dropna(axis=0, thresh=50, inplace=True)

Upvotes: 1

user7864386
user7864386

Reputation:

You could use isna + mean on axis=1 to find the percentage of NaN values for each row. Then select the rows where it's less than 0.9 (i.e. 90%) using loc:

out = df.loc[df.isna().mean(axis=1)<0.9]

Upvotes: 1

Related Questions