How do I use GeneratorExit?

Question

I have the following mcve:

import logging
class MyGenIt(object):
    def __init__(self, name, content):
        self.name = name
        self.content = content
    def __iter__(self):
        with self:
            for o in self.content:
                yield o
    def __enter__(self):
        return self
    def __exit__(self,  exc_type, exc_value, traceback):
        if exc_type:
            logging.error("Aborted %s", self,
                          exc_info=(exc_type, exc_value, traceback))

And here is sample use:

for x in MyGenIt("foo",range(10)):
    if x == 5:
        raise ValueError("got 5")

I would like logging.error to report the ValueError, but instead it reports GeneratorExit:

ERROR:root:Aborted <__main__.MyGenIt object at 0x10ca8e350>
Traceback (most recent call last):
  File "<stdin>", line 8, in __iter__
GeneratorExit

When I catch GeneratorExit in __iter__:

def __iter__(self):
    with self:
        try:
            for o in self.content:
                yield o
        except GeneratorExit:
            return

nothing is logged (of course) because __exit__ is called with exc_type=None.

1. Why do I see GeneratorExit instead of ValueError in __exit__?
2. What do I do to get the desired behavior, i.e., ValueError in __exit__?

Answer 1

Just a quick note that you could "bring the context manager out" of the generator, and by only changing 3 lines get:

import logging
class MyGenIt(object):
    def __init__(self, name, content):
        self.name = name
        self.content = content

    def __iter__(self):
        for o in self.content:
            yield o

    def __enter__(self):
        return self

    def __exit__(self,  exc_type, exc_value, traceback):
        if exc_type:
            logging.error("Aborted %s", self,
                          exc_info=(exc_type, exc_value, traceback))


with MyGenIt("foo", range(10)) as gen:
    for x in gen:
        if x == 5:
            raise ValueError("got 5")

A context manager that could also act as an iterator -- and would catch caller code exceptions like your ValueError.

Answer 2

The basic problem is that you are trying to use a with statement inside the generator to catch an exception that is raised outside the generator. You cannot get __iter__ to see the ValueError, because __iter__ is not executing at the time the ValueError is raised.

The GeneratorExit exception is raised when the generator itself is deleted, which happens when it is garbage collected. As soon as the exception occurs, the for loop terminates; since the only reference to the generator (the object obtained by calling __iter__) is in the loop expression, terminating the loop removes the only reference to the iterator and makes it available for garbage collection. It appears that here it is being garbage collected immediately, meaning that the GeneratorExit exception happens between the raising of the ValueError and the propagation of that ValueError to the enclosing code. The GeneratorExit is normally handled totally internally; you are only seeing it because your with statement is inside the generator itself.

In other words, the flow goes something like this:

1. Exception is raised outside the generator
2. for loop exits
   1. Generator is now available for garbage collection
   2. Generator is garbage collected
      1. Generator's .close() is called
      2. GeneratorExit is raised inside the generator
3. ValueError propagates to calling code

The last step does not occur until after your context manager has seen the GeneratorExit. When I run your code, I see the ValueError raised after the log message is printed.

You can see that the garbage collection is at work, because if you create another reference to the iterator itself, it will keep the iterator alive, so it won't be garbage collected, and so the GeneratorExit won't occur. That is, this "works":

it = iter(MyGenIt("foo",range(10)))
for x in it:
    if x == 5:
        raise ValueError("got 5")

The result is that the ValueError propagates and is visible; no GeneratorExit occurs and nothing is logged. You seem to think that the GeneratorExit is somehow "masking" your ValueError, but it isn't really; it's just an artifact introduced by not keeping any other references to the iterator. The fact that GeneratorExit occurs immediately in your example isn't even guaranteed behavior; it's possible that the iterator might not be garbage-collected until some unknown time in the future, and the GeneratorExit would then be logged at that time.

Turning to your larger question of "why do I see GeneratorExit", the answer is that that is the only exception that actually occurs within the generator function. The ValueError occurs outside the generator, so the generator can't catch it. This means your code can't really work in the way you seem to intend it to. Your with statement is inside the generator function. Thus it can only catch exceptions that happen in the process of yielding items from the generator; there generator has no knowledge of what happens between the times when it advances. But your ValueError is raised in the body of the loop over the generator contents. The generator is not executing at this time; it's just sitting there suspended.

You can't use a with statement in a generator to magically trap exceptions that occur in the code that iterates over the generator. The generator does not "know" about the code that iterates over it and can't handle exceptions that occur there. If you want to catch exceptions within the loop body, you need a separate with statement enclosing the loop itself.

Answer 3

The GeneratorExit is raised whenever a generator or coroutine is closed. Even without the context manager, we can replicate the exact condition with a simple generator function that prints out the exception information when it errors (further reducing the provided code to show exactly how and where that exception is generated).

import sys
def dummy_gen():
    for idx in range(5): 
        try:
            yield idx 
        except:
            print(sys.exc_info())
            raise

for i in dummy_gen():
    raise ValueError('foo')

Usage:

(<class 'GeneratorExit'>, GeneratorExit(), <traceback object at 0x7f96b26b4cc8>)
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ValueError: foo

Note there was also an exception that was raised inside the generator itself, as noted that the except block was executed. Note that the exception was also further raise'd after the print statement but note how that isn't actually shown anywhere, because it is handled internally.

We can also abuse this fact to see if we can manipulate the flow by swallowing the GeneratorExit exception and see what happens. This can be done by removing the raise statement inside the dummy_gen function to get the following output:

(<class 'GeneratorExit'>, GeneratorExit(), <traceback object at 0x7fd1f0438dc8>)
Exception ignored in: <generator object dummy_gen at 0x7fd1f0436518>
RuntimeError: generator ignored GeneratorExit
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ValueError: foo

Note how there is an internal RuntimeError that was raised that complained about the generator ignoring the GeneratorExit function. So we from this we can clearly see that this exception is produced by the generator itself inside the generator function, and the ValueError that is raised outside that scope is never present inside the generator function.

Since a context manager will trap all exceptions as is, and the context manager is inside the generator function, whatever exception raised inside it will simply be passed to __exit__ as is. Consider the following:

class Context(object):
    def __enter__(self):
        return self
    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type:
            logging.error("Aborted %s", self,
                          exc_info=(exc_type, exc_value, traceback))

Modify the dummy_gen to the following:

def dummy_gen():
    with Context():
        for idx in range(5):
            try:
                yield idx
            except:
                print(sys.exc_info())
                raise

Running the resulting code:

(<class 'GeneratorExit'>, GeneratorExit(), <traceback object at 0x7f44b8fb8908>)
ERROR:root:Aborted <__main__.Context object at 0x7f44b9032d30>
Traceback (most recent call last):
  File "foo.py", line 26, in dummy_gen
    yield idx
GeneratorExit
Traceback (most recent call last):
  File "foo.py", line 41, in <module>
    raise ValueError('foo')
ValueError: foo

The same GeneratorExit that is raised is now presented to the context manager, because this is the behavior that was defined.