CODEWITHSUNDEEP
pythonpython-2.7dictionary
shrishinde
shrishinde

Reputation: 3405

How to efficiently filter data from list of dictionaries

I have data in tuple_list and dictionaries_list and in both the lists there can be thousands of records. I want to filter data from dictionaries_list based on records from tuple_list. Currently I wrote following code but it takes lot of time as for each element from list of tuples it has to iterate entire list of dictionaries. 

# tuple_list can be like [(('s', 45), ('t', 30)), (('s', 5), ('t', 3))]
for target_tuple in tuple_list:
    # target_tuple can have data like (('s', 45), ('t', 30))
    # dictionaries_list can have data like [{'a': 5, 's': 45, 't': 10}, {}, {}]
    if some_parameter == 'something':
        m1_dicts = [d for d in dictionaries_list if d['s'] == target_tuple[0][1]]
    else:
        m1_dicts = [d for d in dictionaries_list if d['t'] == target_tuple[0][1]]

Please suggest some way to improve this.

Upvotes: 3

Views: 101

Answers (1)

Skycc
Skycc

Reputation: 3555

From what i understand, you wanna keep the dictionary if any of the tuple_list found match within the dictionary. It did improve a bit on runtime because it only run through tuple_list once, but might not the best optimize yet as it still need to iterate through list of dictionaries

# taking small dataset as example
tuple_list=[(('s', 45), ('t', 30)), (('s', 5), ('t', 6))]
dictionaries_list = [{'a': 5, 's': 45, 't': 10}, {'s':5,'t':6}, {'a':2,'s':22,'t':30}]

tuple_dict = {}
tuple_dict['s'] = [i[0][1] for i in tuple_list]
tuple_dict['t'] = [i[1][1] for i in tuple_list]
if some_parameter == 'something':
    # check for 's'
    m1_dicts = [d for d in dictionaries_list if d['s'] in tuple_dict['s']]
    # m1_dicts = [{'a': 5, 's': 45, 't': 10}, {'s':5,'t':6}]
else:
    # check for 't'
    m1_dicts = [d for d in dictionaries_list if d['t'] in tuple_dict['t']]
    # m1_dicts = [{'s':5,'t':6}, {'a':2,'s':22,'t':30}]

Instead of some_parameter = 'something', suggest to use some_patameter as the key to check for

some_parameter = 's' # check for 's'
m1_dicts = [d for d in dictionaries_list if d[some_parameter] in tuple_dict[some_parameter]]

Upvotes: 1

Related Questions