CODEWITHSUNDEEP
pythonpython-3.xgenerator
Rusty
Rusty

Reputation: 142

`map` causing infinite loop in Python 3

I have the following code:

def my_zip(*iterables):
    iterators = tuple(map(iter, iterables))
    while True:
        yield tuple(map(next, iterators))

When my_zip is called, it just creates an infinite loop and never terminates. If I insert a print statement (like shown below), it is revealed that my_zip is infinitely yielding empty tuples!

def my_zip(*iterables):
    iterators = tuple(map(iter, iterables))
    while True:
        t = tuple(map(next, iterators))
        print(t)
        yield t

However, the equivalent code with a generator expression works fine:

def my_genexp_zip(*iterables):
    iterators = tuple(iter(it) for it in iterables)
    while True:
        try:
            yield tuple(next(it) for it in iterators)
        except:
            print("exception caught!")
            return

Why is the function with map not behaving as expected? (Or, if it is expected behavior, how could I modify its behavior to match that of the function using the generator expression?)

I am testing with the following code:

print(list(my_genexp_zip(range(5), range(0, 10, 2))))
print(list(my_zip(range(5), range(0, 10, 2))))

Upvotes: 5

Views: 172

Answers (2)

no comment
no comment

Reputation: 10465

Your map version has an explicit infinite loop. The only way it might stop is if the body raised an exception. But the map iterator lets the StopIteration from the failed next call fall through, and then tuple simply thinks map raised it and stops (resulting in empty tuples as soon as the first iterator is exhausted).

That's all expected, the roundrobin itertools recipe for example also uses this trick:

def roundrobin(*iterables):
    "Visit input iterables in a cycle until each is exhausted."
    # roundrobin('ABC', 'D', 'EF') → A D E B F C
    # Algorithm credited to George Sakkis
    iterators = map(iter, iterables)
    for num_active in range(len(iterables), 0, -1):
        iterators = cycle(islice(iterators, num_active))
        yield from map(next, iterators)

Here, each failed next call simply causes the yield from to stop, and then the algorithm continues with the remaining iterators (as long as any are still active).

You could fix your map version for example by checking the tuple length and breaking if it's shorter than iterables. Or you could chain a RuntimeError-raising iterator onto all iterables and catch that just like in your genexp version:

from itertools import chain, repeat

def my_fixed_zip(*iterables):
    def error():
        raise RuntimeError
        yield
    iterators = tuple(map(chain, iterables, repeat(error())))
    while True:
        try:
            yield tuple(map(next, iterators))
        except:
            print("exception caught!")
            return

Attempt This Online!

Upvotes: 1

blhsing
blhsing

Reputation: 107015

The two pieces of code you provided are not actually "equivalent", with the function using generator expressions notably having a catch-all exception handler around the generator expression producing items for tuple output.

And if you actually make the two functions "equivalent" by removing the exception handler:

def my_listcomp_zip(*iterables):
    iterators = tuple(iter(it) for it in iterables)
    while True:
        yield tuple(next(it) for it in iterators)

print(list(my_listcomp_zip(range(5), range(0, 10, 2))))

you'll get a traceback of:

Traceback (most recent call last):
  File "test.py", line 4, in <genexpr>
    yield tuple(next(it) for it in iterators)
                ~~~~^^^^
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(list(my_listcomp_zip(range(5), range(0, 10, 2))))
          ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "test.py", line 4, in my_listcomp_zip
    yield tuple(next(it) for it in iterators)
          ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: generator raised StopIteration

So it is clear by now that the reason why your infinite loop with while True: can end at all with your generator expression version of the function is because a RuntimeError is caught by your catch-all exception handler, which returns from the function.

And this is because since Python 3.7, with the implementation of PEP-479, StopIteration raised inside a generator gets automatically turned into a RuntimeError in order not to be confused with the StopIteration raised by an exhausted generator itself.

If you try your code in an earlier Python version (such as 2.7), you'll find the generator expression version of the function gets stuck in the infinite loop just as well, where the StopIteration exception raised by next bubbles out from the generator and gets handled by the tuple constructor to produce an empty tuple, just like the map version of your function. And addressing this exception masking effect is exactly why PEP-479 was proposed and implemented.

Upvotes: 3

Related Questions