CODEWITHSUNDEEP
pythonpython-3.xgeneratorgenerator-expression
amwinter
amwinter

Reputation: 3139

next() doesn't play nice with any/all in python

I ran down a bug today that came about because I was using next() to extract a value, and 'not found' emits a StopIteration.

Normally that would halt the program, but the function using next was being called inside an all() iteration, so the all just terminated early and returned True.

Is this an expected behavior? Are there style guides that help avoid this kind of thing?

Simplified example:

def error(): return next(i for i in range(3) if i==10)
error() # fails with StopIteration
all(error() for i in range(2)) # returns True

Upvotes: 26

Views: 1896

Answers (2)

schesis
schesis

Reputation: 59118

While this is the default behaviour in Python versions up to and including 3.6, it's considered to be a mistake in the language, and is scheduled to change in Python 3.7 so that an exception is raised instead.

As PEP 479 says:

The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.

From Python 3.5 onwards, it's possible to change the default behaviour to that scheduled for 3.7. This code:

# gs_exc.py

from __future__ import generator_stop

def error():
    return next(i for i in range(3) if i==10)

all(error() for i in range(2))

… raises the following exception:

Traceback (most recent call last):
  File "gs_exc.py", line 8, in <genexpr>
    all(error() for i in range(2))
  File "gs_exc.py", line 6, in error
    return next(i for i in range(3) if i==10)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "gs_exc.py", line 8, in <module>
    all(error() for i in range(2))
RuntimeError: generator raised StopIteration

In Python 3.5 and 3.6 without the __future__ import, a warning is raised. For example:

# gs_warn.py

def error():
    return next(i for i in range(3) if i==10)

all(error() for i in range(2))


$ python3.5 -Wd gs_warn.py 
gs_warn.py:6: PendingDeprecationWarning: generator '<genexpr>' raised StopIteration
  all(error() for i in range(2))


$ python3.6 -Wd gs_warn.py 
gs_warn.py:6: DeprecationWarning: generator '<genexpr>' raised StopIteration
  all(error() for i in range(2))

Upvotes: 24

Mark Ransom
Mark Ransom

Reputation: 308138

The problem isn't in using all, it's that you have a generator expression as the parameter to all. The StopIteration gets propagated to the generator expression, which doesn't really know where it originated, so it does the usual thing and ends the iteration.

You can see this by replacing your error function with something that raises the error directly:

def error2(): raise StopIteration

>>> all(error2() for i in range(2))
True

The final piece of the puzzle is knowing what all does with an empty sequence:

>>> all([])
True

If you're going to use next directly, you should be prepared to catch StopIteration yourself.

Edit: Nice to see that the Python developers consider this a bug and are taking steps to change it in 3.7.

Upvotes: 9

Related Questions