CODEWITHSUNDEEP
pythonpandasdataframe
Ryan
Ryan

Reputation: 149

Python - Pandas - Key Error during dropna call for specific subsets

My goal: I wish to drop rows who have NaN in specific columns. I will allow NaN to exist on some columns but not others. English Example: If value of 'detail_age' in a row is NaN, I want to remove that row.

Here is a view of my data:

import pandas as pd
df = pd.read_csv('allDeaths.csv', index_col=0, nrows=3, engine='python')
print(df.shape)
print(list(df))

Which outputs:

(3,15)
['education_1989_revision', 'education_2003_revision', 
'education_reporting_flag', 'sex', 'detail_age', 'marital_status', 
'current_data_year', 'injury_at_work', 'manner_of_death', 'activity_code', 
'place_of_injury_for_causes_w00_y34_except_y06_and_y07_', '358_cause_recode', 
'113_cause_recode', '39_cause_recode', 'race']

When I attempt to remove rows who's columns value is NaN with the following:

df.dropna(subset=[2,3,4,5,6,7,8,9,11,12,13,14], axis=1, inplace=True, how='any')

I get the following error:

Traceback (most recent call last):
  File "clean.py", line 10, in <module>
    df.dropna(subset=[2,3,4,5,6,7,8,9,11,12,13,14], axis=1, inplace=True, how='any')
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py", line 3052, in dropna
    raise KeyError(list(np.compress(check, subset)))
KeyError: [3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14]

Which is weird because this works:

df.dropna(subset=[2], axis=1, inplace=True, how='any')

But not this:

df.dropna(subset=[5], axis=1, inplace=True, how='any')

So there must be something wrong with certain columns or the values in those columns. Here is a peek at my data with df.head(3):

As image because formatting is annoying

Upvotes: 4

Views: 15282

Answers (1)

MaxU - stand with Ukraine
MaxU - stand with Ukraine

Reputation: 210852

Demo:

In [360]: df
Out[360]:
      A     B     C   D
0   1.0   2.0   NaN   4
1   5.0   NaN   7.0   8
2   NaN  10.0  11.0  12
3  13.0  14.0  15.0  16

In [362]: df = df.dropna(subset=df.columns[[1,2]], how='any')

In [363]: df
Out[363]:
      A     B     C   D
2   NaN  10.0  11.0  12
3  13.0  14.0  15.0  16

PS of course you can specify column names instead:

In [370]: df.dropna(subset=['B','C'], how='any')
Out[370]:
      A     B     C   D
2   NaN  10.0  11.0  12
3  13.0  14.0  15.0  16

Upvotes: 6

Related Questions