CODEWITHSUNDEEP
pythonpython-3.xgenerator
max
max

Reputation: 52253

Why exhausted generators raise StopIteration more than once?

Why is it that when an exhausted generator is called several times, StopIteration is raised every time, rather than just on the first attempt? Aren't subsequent calls meaningless, and indicate a likely bug in the caller's code?

def gen_func():
    yield 1
    yield 2
gen = gen_func()
next(gen)
next(gen)
next(gen) # StopIteration as expected
next(gen) # why StopIteration and not something to warn me that I'm doing something wrong

This also results in this behavior when someone accidentally uses an expired generator:

def do_work(gen):
    for x in gen:
        # do stuff with x
        pass

    # here I forgot that I already used up gen
    # so the loop does nothing without raising any exception or warning
    for x in gen:
        # do stuff with x
        pass

def gen_func():
    yield 1
    yield 2

gen = gen_func()
do_work(gen)

If second and later attempts to call an exhausted generator raised a different exception, it would have been easier to catch this type of bugs.

Perhaps there's an important use case for calling exhausted generators multiple times and getting StopIteration?

Upvotes: 6

Views: 1166

Answers (3)

Facundo
Facundo

Reputation: 160

Here's an implementation of a wrapper that raises an error whenever StopIteration is raised more than once, as already noted by VPfB, this is implementation is considered broken

#!/usr/bin/env python3.8
from typing import TypeVar, Iterator

"""
https://docs.python.org/3/library/stdtypes.html#iterator-types
This is considered broken by the iterator protocol, god knows why
"""

class IteratorExhaustedError(Exception):
    """Exception raised when exhausted iterators are ``next``d"""

T = TypeVar("T")
class reuse_guard(Iterator[T]):
    """
    Wraps an iterator so that StopIteration is only raised once,
    after that, ``IteratorExhaustedError`` will be raised to detect
    fixed-size iterator misuses
    """
    def __init__(self, iterator: Iterator[T]):
        self._iterated: bool = False
        self._iterator = iterator

    def __next__(self) -> T:
        try:
            return next(self._iterator)
        except StopIteration as e:
            if self._iterated:
                raise IteratorExhaustedError(
                    "This iterator has already reached its end")
            self._iterated = True
            raise e
    
    def __iter__(self) -> Iterator[T]:
        return self

Example:

In [48]: iterator = reuse_guard(iter((1, 2, 3, 4)))

In [49]: list(iterator)
Out[49]: [1, 2, 3, 4]

In [50]: list(iterator)
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-47-456650faec86> in __next__(self)
     19         try:
---> 20             return next(self._iterator)
     21         except StopIteration as e:

StopIteration:

During handling of the above exception, another exception occurred:

IteratorExhaustedError                    Traceback (most recent call last)
<ipython-input-50-5070d0fe4365> in <module>
----> 1 list(iterator)

<ipython-input-47-456650faec86> in __next__(self)
     21         except StopIteration as e:
     22             if self._iterated:
---> 23                 raise IteratorExhaustedError(
     24                     "This iterator has already reached its end")
     25             self._iterated = True

IteratorExhaustedError: This iterator has already reached its end

Edit: After revisiting the documentation on the iterator protocol it seems to me that the purpose of stating that iterators that do not continue to raise StopIteration should be considered broken is aimed more at the iterators that yield values instead of raising exceptions, that in this case make it more clear that the iterator should not be used once it's been exhausted. This is merely my interpretation thought.

Upvotes: 0

VPfB
VPfB

Reputation: 17267

It is a part of the iteration protocol:

Once an iterator’s __next__() method raises StopIteration, it must continue to do so on subsequent calls. Implementations that do not obey this property are deemed broken.

Source: https://docs.python.org/3/library/stdtypes.html#iterator-types

Upvotes: 4

user2357112
user2357112

Reputation: 280778

Perhaps there's an important use case for calling exhausted generators multiple times and getting StopIteration?

There is, specifically, when you want to perform multiple loops on the same iterator. Here's an example from the itertools docs that relies on this behavior:

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

Upvotes: 4

Related Questions