CODEWITHSUNDEEP
pythonpandas
Sander
Sander

Reputation: 435

Pandas fillna to empty dictionary

I have a pandas dataframe with a 'metadata' column that should contain a dictionary as value. However, some values are missing and set to NaN. I would like this to be {} instead. Sometimes, the entire column is missing and initializing it to {} is also problematic.

For adding the column 

tspd['metadata'] = {} # fails
tspd['metadata'] = [{} for _ in tspd.index] # works

For filling missing values

tspd['metadata'].replace(np.nan,{}) # does nothing
tspd['metadata'].fillna({})  # likewise does nothing
tspd.loc[tspd['metadata'].isna(), 'metadata'] = {} # error
tspd['metadata'] = tspd['metadata'].where(~tspd['metadata'].isna(), other={}) # this sets the NaN values to <built-in method values of dict object>

So adding the column works, but is a bit ugly. Replacing the values without some (slow) loop seems not possible.

Upvotes: 4

Views: 2953

Answers (2)

jezrael
jezrael

Reputation: 863226

You can use np.nan == np.nan is False, so for replace missing values is possible use:

tspd = pd.DataFrame({'a': [0,1,2], 'metadata':[{'a':'s'}, np.nan, {'d':'e'}]})

tspd['metadata'] = tspd['metadata'].apply(lambda x: {} if x != x else x)
print(tspd)

   a    metadata
0  0  {'a': 's'}
1  1          {}
2  2  {'d': 'e'}

Or:

tspd['metadata'] = [{} if x != x else x for x in tspd['metadata']]

Upvotes: 8

BENY
BENY

Reputation: 323326

Do not using [{}] * len(tspd)

tspd['metadata'] = [{}for x in range(len(tspd))]
tspd
Out[326]: 
   a metadata
0  0       {}
1  1       {}
2  2       {}

Detail 

tspd['metadata'] = [{}] * len(tspd)
tspd['metadata'].iloc[0]['lll']=1
tspd # see all duplicated here ,since they are the same copy 
Out[324]: 
   a    metadata
0  0  {'lll': 1}
1  1  {'lll': 1}
2  2  {'lll': 1}

Do it one by one , each time create the iid {}

tspd['metadata'] = [{}for x in range(len(tspd))]
tspd
Out[326]: 
   a metadata
0  0       {}
1  1       {}
2  2       {}
tspd['metadata'].iloc[0]['lll']=1
tspd
Out[328]: 
   a    metadata
0  0  {'lll': 1}
1  1          {}
2  2          {}

Upvotes: 2

Related Questions