CODEWITHSUNDEEP
python
Wells
Wells

Reputation: 10969

python dict/list comprehension: why is it slower than for loop?

Consider that events here has about 48,000 dictionary objects:

keyed_events = { gid: [ r for r in events if r['gid'] == gid ] for gid in gidlist }

Is about 4x slower than:

keyed_events = {}
for event in events:
    gid = event['gid']
    if gid not in keyed_events:
        keyed_events[gid] = []
    keyed_events[gid].append(event)

Though the former looks more efficient. Why is it slower? Iterating through events each pass of the dict comprehension?

Upvotes: 1

Views: 439

Answers (3)

DaoWen
DaoWen

Reputation: 33019

Your dictionary + list comprehension more closely matches this code:

keyed_events = {}
for gid in gidlist:
    for r in events:
        if r['gid'] == gid:
            keyed_events[gid].append(r)

Notice that the loop is doubly nested. You loop over all of the events for each gid.

A closer match to your iterative code would be this:

keyed_events = itertools.groupby(events, 'gid')

Upvotes: 3

mehdy
mehdy

Reputation: 3374

I think it's because in the snippet you're iterating through both gidlist and events but in the first method but iterating through just events.

totally I think the inner inline for is inefficient

Upvotes: 0

Ayush
Ayush

Reputation: 3935

keyed_events = { gid: [ r for r in events if r['gid'] == gid ] for gid in gidlist }

The list/dict comprehension runs for len(gidlist)*len(events) number of times as it loops over events inside of a loop over gidlist.

The for loop, on the other hand, has just a single for loop over events with a gid not in keyed_events which is a O(1) operation

Upvotes: 4

Related Questions