CODEWITHSUNDEEP
pythonpython-3.xlistloopsdictionary
user12983086
user12983086

Reputation:

Python : How to remove rows which has nan values for dict inside list

Have a dict in list with values:

a:

[{'name': 'toom D',
  'status_ref': True,
  'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
  'expiry': None,
  'renew': nan},
 {'name': 'russ arnold',
  'status_ref': True,
  'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
  'expiry': None,
  'renew': nan},
 {'name': 'cart wright',
  'status_ref': True,
  'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
  'expiry': 'no',
  'renew': 'yes'}]

How to remove whole rows for records having nan , None , NaN values in list(dict)

expected_output:

b:

[{'name': 'toom D',
'status_ref': True,
'date_status_ref': Timestamp('2018-12-17 09:29:58.864000')
},
{'name': 'russ arnold',
'status_ref': True,
'date_status_ref': Timestamp('2018-12-17 09:29:58.864000')
},
{'name': 'cart wright',
'status_ref': True,
'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
'expiry': 'no',
'renew': 'yes'}]

Upvotes: 3

Views: 92

Answers (3)

Pygirl
Pygirl

Reputation: 13349

Not the best solution but you can try.

a = [{'name': 'toom D',
  'status_ref': True,
  'date_status_ref': 'a',
  'expiry': None,
  'renew': nan},
 {'name': 'russ arnold',
  'status_ref': True,
  'date_status_ref': 'b',
  'expiry': None,
  'renew': nan},
 {'name': 'cart wright',
  'status_ref': True,
  'date_status_ref': 'c',
  'expiry': 'no',
  'renew': 'yes'}]

for d in a:
    for k,v in d.copy().items():
        if (str(v) in ['None', 'NaN', 'nan']):
            d.pop(k)

a: 

[{'name': 'toom D', 'status_ref': True, 'date_status_ref': 'a'},
 {'name': 'russ arnold', 'status_ref': True, 'date_status_ref': 'b'},
 {'name': 'cart wright',
  'status_ref': True,
  'date_status_ref': 'c',
  'expiry': 'no',
  'renew': 'yes'}]

Upvotes: 0

Amir
Amir

Reputation: 1905

You can iterate through key/values of your dicts and filter them. like this:

a = [{'name': 'toom D',
    'status_ref': True,
    'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
    'expiry': None,
    'renew': nan},
    {'name': 'russ arnold',
    'status_ref': True,
    'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
    'expiry': None,
    'renew': nan},
    {'name': 'cart wright',
    'status_ref': True,
    'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
    'expiry': 'no',
    'renew': 'yes'}]

def filter_dict(d):
    new_dict = dict()
    for (key, value) in d.items():
        if not (value is None or np.isnan(value)):
            new_dict[key] = value
    return new_dict

b = [filter_dict(d) for d in a]

Upvotes: 0

Mario Ishac
Mario Ishac

Reputation: 5907

Assuming nan is math.nan, you can do this:

for sub_a in a:
    for key, val in sub_a.copy().items():
        try:
            if val is None or math.isnan(val):
                del sub_a[key]
        except TypeError:
            continue

The above will mutate the original a. If you want a separate dict b as the output, you can do b = a.copy() and replace a with b in the outer for.

The reason I wrapped math.isnan in a try / except is because not all values in your dicts are real numbers, some are strings. If the TypeError is thrown, we know that the value couldn't possibly be nan.

The resulting dict is:

[{'name': 'toom D',
  'status_ref': True,
  'date_status_ref': Timestamp('2018-12-17 09:29:58.864000')},
 {'name': 'russ arnold',
  'status_ref': True,
  'date_status_ref': Timestamp('2018-12-17 09:29:58.864000')},
 {'name': 'cart wright',
  'status_ref': True,
  'date_status_ref': Timestamp('2018-12-17 09:29:58.864000'),
  'expiry': 'no',
  'renew': 'yes'}]

Upvotes: 3

Related Questions