CODEWITHSUNDEEP
pythonpython-3.xiteratorgeneratoriterable-unpacking
Yngve Moe
Yngve Moe

Reputation: 1167

Weird behaviour of generator unpacking in Python

I am currently trying to be more familiar with iterators in Python, and I encountered some weird behaviour. Essentially, I get the wrong behaviour with a generator comprehension, but the correct behaviour with a list comprehension.

Let me start by explaining what I try to do, and then what behaviour I get.
Imagine having a dictionary of iterables, e.g.

d = {'a': [1, 2, 3], 'b': [4, 5]}

What I want is to have a list of dictionaries with all possible combinations of the iterable. For the first example, this would be

l = [
        {'a': 1, 'b': 4},
        {'a': 1, 'b': 5},
        {'a': 2, 'b': 4},
        {'a': 2, 'b': 5},
        {'a': 3, 'b': 4},
        {'a': 3, 'b': 5},
    ]

To do this, I created this generator:

def dict_value_iterator(d):
    for k, v in d.items():         
        yield ((k, vi) for vi in v)

The idea was to run the following code to get the wanted result

def get_all_dicts(d):
    return map(dict, *itertools.product(dict_value_iterator(d)))

Now, for the strange behaviour.
To test that the dict_value_iterator generator indeed did what I hoped it would, I ran the following code:

for i in dict_value_iterator(d):
    print(list(i))

which indeed does what I hoped it would, namely print out the following:

[('a', 1), ('a', 2), ('a', 3)]
[('b', 4), ('b', 5)]

However, when I run the following code

def test_unpacking(*args):
    for a in args:
        print(list(a))
test_unpacking(*dict_value_iterator(d))

I get the output

[('b', 1), ('b', 2), ('b', 3)]
[('b', 4), ('b', 5)]

This makes little to no sense for me, why does iterator unpacking change anything.

Final note.
The way I found it was by running the get_all_dicts function on d, which resulted in the following output

[{'b': 4}, {'b': 5}, {'b': 4}, {'b': 5}, {'b': 4}, {'b': 5}]

However, when I modify the dict_value_iterator as follows

def dict_value_iterator(d):
    for k, v in d.items():         
        yield ((k, vi) for vi in v)

I get this output

[{'a': 1, 'b': 4},
 {'a': 1, 'b': 5},
 {'a': 2, 'b': 4},
 {'a': 2, 'b': 5},
 {'a': 3, 'b': 4},
 {'a': 3, 'b': 5}]

which is what I want.

Upvotes: 3

Views: 173

Answers (1)

Ry-
Ry-

Reputation: 224942

Here’s a simplified version:

generators = []

for i in [1, 2]:
    generators.append((i for _ in [1]))

print(list(generators[0]))  # [2]

Only a single variable called i exists, and the for loop sets it repeatedly. All generators created by the generator expression refer to the same i and don’t read it until the loop has exited.

One way to fix it is by creating another scope with a function (like you would in ES5, for example):

def dict_value_iterator(d):
    def get_generator(k, v):
        return ((k, vi) for vi in v)

    for k, v in d.items():         
        yield get_generator(k, v)

Upvotes: 2

Related Questions