Extract elements from complex list of lists and dictionaries in python

Question

I have a list with a number of listed lists and dictionaries representing NYC subway cars:

[[{'arrival': {'time': 1506873749L},
   'departure': {'time': 1506873749L},
   'schedule_relationship': 0,
   'stop_id': u'B20S'},
  {'arrival': {'time': 1506873854L},
   'departure': {'time': 1506873854L},
   'schedule_relationship': 0,
   'stop_id': u'B21S'},
  {'arrival': {'time': 1506873989L},
   'departure': {'time': 1506873989L},
   'schedule_relationship': 0,
   'stop_id': u'B22S'},
  {'arrival': {'time': 1506874184L},
   'departure': {'time': 1506874184L},
   'schedule_relationship': 0,
   'stop_id': u'B23S'},
  {'arrival': {'time': 1506874469L},
   'departure': {'time': 1506874469L},
   'schedule_relationship': 0,
   'stop_id': u'D43S'}],
 [{'arrival': {'time': 1506873814L},
   'departure': {'time': 1506873814L},
   'schedule_relationship': 0,
   'stop_id': u'D10N'},
  {'arrival': {'time': 1506873877L},
   'departure': {'time': 1506873877L},
   'schedule_relationship': 0,
   'stop_id': u'D09N'},
  {'arrival': {'time': 1506873997L},
   'departure': {'time': 1506873997L},
   'schedule_relationship': 0,
   'stop_id': u'D08N'},
  {'arrival': {'time': 1506874087L},
   'departure': {'time': 1506874087L},
   'schedule_relationship': 0,
   'stop_id': u'D07N'},
  {'arrival': {'time': 1506874177L},
   'departure': {'time': 1506874177L},
   'schedule_relationship': 0,
   'stop_id': u'D06N'},
  {'arrival': {'time': 1506874267L},
   'departure': {'time': 1506874267L},
   'schedule_relationship': 0,
   'stop_id': u'D05N'},
  {'arrival': {'time': 1506874357L},
   'departure': {'time': 1506874357L},
   'schedule_relationship': 0,
   'stop_id': u'D04N'},
  {'arrival': {'time': 1506874477L},
   'departure': {'time': 1506874477L},
   'schedule_relationship': 0,
   'stop_id': u'D03N'},
  {'arrival': {'time': 1506874627L},
   'departure': {'time': 1506874627L},
   'schedule_relationship': 0,
   'stop_id': u'D01N'}]]

I am trying to identify the entries associated with a specific stop_id. For example, if I was searching for 'D03N' I would like to return the entire entry associated with it:

 {'arrival': {'time': 1506874477L},
       'departure': {'time': 1506874477L},
       'schedule_relationship': 0,
       'stop_id': u'D03N'}

Unfortunately, whenever I try and use the suggestions from this answer: Python list of dictionaries search I end up with a 'TypeError: list indices must be integers, not str' error message. I'm not sure if this is because I am implementing that solution incorrectly or the solution does not apply because of the relative complexity of this list compared to the one in the original question.

Is there a way to pluck specific entries out of this list?

Answer 1

Here is a recursive solution that works with any level of nested lists. This function searches (DFS) the list like it's a graph where your list is the root node, sub lists are parent nodes and dictionaries are leafs nodes.

def find_by_stopid(at, target, saveto):
    if isinstance(at, dict):
        if at['stop_id'] == target:
            saveto.append(at)
        return

    for x in at:
        find_by_stopid(x, target, saveto)

found = []
target = u'D03N'

# data is the list you have, targets is the string to match
# and found is where matches are saved
find_by_stopid(data, target, found)

print(found)

Answer 2

You can try this:

entry = 'D03N'
final_entries = [[b for b in i if b["stop_id"] == entry] for i in entry_data]
try:
   new_final_entries = [i for i in final_entries if i][0][0]
except:
   print("Entry not found")

Where entry data is the full dictionary posted in the original question.

Output:

{'arrival': {'time': 1506874477L}, 'schedule_relationship': 0, 'departure': {'time': 1506874477L}, 'stop_id': u'D03N'}

Answer 3

>>> from itertools import chain
>>> data = [[{'arrival': {'time': 1506873749L}, 'departure': {'time': 1506873749L}, 'schedule_relationship': 0, 'stop_id': u'B20S'}, {'arrival': {'time': 1506873854L}, 'departure': {'time': 1506873854L}, 'schedule_relationship': 0, 'stop_id': u'B21S'}, {'arrival': {'time': 1506873989L}, 'departure': {'time': 1506873989L}, 'schedule_relationship': 0, 'stop_id': u'B22S'}, {'arrival': {'time': 1506874184L}, 'departure': {'time': 1506874184L}, 'schedule_relationship': 0, 'stop_id': u'B23S'}, {'arrival': {'time': 1506874469L}, 'departure': {'time': 1506874469L}, 'schedule_relationship': 0, 'stop_id': u'D43S'}], [{'arrival': {'time': 1506873814L}, 'departure': {'time': 1506873814L}, 'schedule_relationship': 0, 'stop_id': u'D10N'}, {'arrival': {'time': 1506873877L}, 'departure': {'time': 1506873877L}, 'schedule_relationship': 0, 'stop_id': u'D09N'}, {'arrival': {'time': 1506873997L}, 'departure': {'time': 1506873997L}, 'schedule_relationship': 0, 'stop_id': u'D08N'}, {'arrival': {'time': 1506874087L}, 'departure': {'time': 1506874087L}, 'schedule_relationship': 0, 'stop_id': u'D07N'}, {'arrival': {'time': 1506874177L}, 'departure': {'time': 1506874177L}, 'schedule_relationship': 0, 'stop_id': u'D06N'}, {'arrival': {'time': 1506874267L}, 'departure': {'time': 1506874267L}, 'schedule_relationship': 0, 'stop_id': u'D05N'}, {'arrival': {'time': 1506874357L}, 'departure': {'time': 1506874357L}, 'schedule_relationship': 0, 'stop_id': u'D04N'}, {'arrival': {'time': 1506874477L}, 'departure': {'time': 1506874477L}, 'schedule_relationship': 0, 'stop_id': u'D03N'}, {'arrival': {'time': 1506874627L}, 'departure': {'time': 1506874627L}, 'schedule_relationship': 0, 'stop_id': u'D01N'}]]

>>> def find(s):
        found = [x for x in chain(*data) if x['stop_id']==s]
        return found[0] if found else None

>>> find(u'D03N')
{'arrival': {'time': 1506874477L}, 'schedule_relationship': 0, 'departure': {'time': 1506874477L}, 'stop_id': u'D03N'}

Answer 4

l = <your list>
[ i for i in sum(l,[]) if i['stop_id'] == 'D03N' ]

or more efficient way

from itertools import chain
[ i for i in chain.from_iterable(l) if i['stop_id'] == 'D03N' ]