CODEWITHSUNDEEP
pythonpandas
frazman
frazman

Reputation: 33213

Unable to weed out NaN rows from dataframe

So, I am trying to clean a dataframe containing some NaN values

I tried all the suggested methods but seems like I cant get rid of NaN.

df = pd.read_csv('filename.tsv', delimiter='\t')
df = df[pd.notnull(df)]
df = df.dropna()

df[pd.isnull(df)]
# gives our records containing NaN (alot of them.)

I am not sure what am I missing?

Edit: The one giving NaN have all the columns as NaN enter image description here

Some more edits: When I try to see the type

heads =  df[df.isnull()].head()
for idx, row in heads.iterrows():
    print idx, type(row.listener_id)

This return 

0 <type 'float'>
1 <type 'float'>
2 <type 'float'>
3 <type 'float'>
4 <type 'float'>

Upvotes: 1

Views: 115

Answers (1)

jezrael
jezrael

Reputation: 862406

I think if need use boolean indexing:

df = df[~df.isnull().any(axis=1)]

But better is use only:

df = df.dropna()

Sample:

df = pd.DataFrame({'A':[np.nan,5,4,5,5,np.nan],
                   'B':[7,8,9,4,2,np.nan],
                   'C':[1,3,5,7,1,np.nan],
                   'D':[5,3,6,9,2,np.nan]})

print (df)
     A    B    C    D
0  NaN  7.0  1.0  5.0
1  5.0  8.0  3.0  3.0
2  4.0  9.0  5.0  6.0
3  5.0  4.0  7.0  9.0
4  5.0  2.0  1.0  2.0
5  NaN  NaN  NaN  NaN

#get True for NaN
print (df.isnull())
       A      B      C      D
0   True  False  False  False
1  False  False  False  False
2  False  False  False  False
3  False  False  False  False
4  False  False  False  False
5   True   True   True   True

#check at least one True per row
print (df.isnull().any(axis=1))
0     True
1    False
2    False
3    False
4    False
5     True
dtype: bool

#boolen indexing with inverting `~` (need select NO NaN rows)
print (df[~df.isnull().any(axis=1)])
     A    B    C    D
1  5.0  8.0  3.0  3.0
2  4.0  9.0  5.0  6.0
3  5.0  4.0  7.0  9.0
4  5.0  2.0  1.0  2.0

#get True for not NaN
print (df.notnull())
       A      B      C      D
0  False   True   True   True
1   True   True   True   True
2   True   True   True   True
3   True   True   True   True
4   True   True   True   True
5  False  False  False  False

#get True if all values per row are True 
print (df.notnull().all(axis=1))
0    False
1     True
2     True
3     True
4     True
5    False
dtype: bool

#boolean indexing
print (df[df.notnull().all(axis=1)])
     A    B    C    D
1  5.0  8.0  3.0  3.0
2  4.0  9.0  5.0  6.0
3  5.0  4.0  7.0  9.0
4  5.0  2.0  1.0  2.0

#simpliest solution
print (df.dropna())
     A    B    C    D
1  5.0  8.0  3.0  3.0
2  4.0  9.0  5.0  6.0
3  5.0  4.0  7.0  9.0
4  5.0  2.0  1.0  2.0

Upvotes: 1

Related Questions