Behaviour of try finally with a generator

Question

I'm following the Effective Python course by Brett Slatkin on O'Reilly: https://learning.oreilly.com/videos/effective-python/9780134175249/

I've come across the following code snippet while learning about closable queues to concurrently handle synchronous actions like downloading/uploading/resizing data:

class ClosableQueue(Queue):
    SENTINEL = object()

    def close(self):
        self.put(self.SENTINEL)

    def __iter__(self):
        while True:
            try:
                item = self.get()

                if item is self.SENTINEL:
                    return 
                else:
                    yield item
            finally:
                self.task_done()

In the snippet above, the finally block is executed after every item is yielded from the queue.

However if we modify the iter function as follows:

class ClosableQueue(Queue):
    SENTINEL = object()
    
    def close(self):
        self.put(self.SENTINEL)

    def __iter__(self):
        try:
            item = self.get()

            if item is self.SENTINEL:
                return 
            else:
                yield item
        finally:
            self.task_done()

Then the finally block is executed after the iterator is exhausted, not after every yield.

I'm not able to clearly understand why this works as described. One possible explanation I came across while searching for an explanation is that the finally block for a generator is supposed to be executed when the generator is exhausted, however, since we have a while True in the first version, the generator is in theory always yielding values, so it is never exhausted, so the finally is run after each yield.

Please let me know if my understanding is correct, if not please provide a better explanation for the behaviour.

Thanks.

Answer 1

def generator_example():
        try:
            while True:
                print("Before yield")
                yield
                print("After yield")
        finally:
            print("In finally")

gen = generator_example()

next(gen)  # Outputs: Before yield
next(gen)  # Outputs: After yield\nBefore yield

The behaviour is not unexpected, Mark more comprehensively answers why.

In your code, after you yield and re-enter, the try's scope ends before the next iteration of the while loop starts.

In the update version, it won't enter the finally because the while loop never exits the original try block. Normally, we could expect a finite iterator to raise StopIteration and break out of the loop. in your case you'd need to break out of your while loop conditionally, ex:

def generator_example():
        counter=0
        try:
            while True:
                if counter >= 2:
                     print("Breaking!")
                     break
                counter += 1
                print("Before yield")
                yield
                print("After yield")
        finally:
            print("In finally")

gen = generator_example()

next(gen)  # same
next(gen)  
next(gen)  # "In finally" printed then StopIteration raised by generator implicit return

About your second example, iteration ends after the first yield for me, same for the Temp class you asked about, consider this modification again:

from queue import Queue


class ClosableQueue(Queue):
    SENTINEL = object()

    def close(self):
        self.put(self.SENTINEL)

    def __iter__(self):
        try:
            while (item := self.get()) != self.SENTINEL:
                yield item
        finally:
            self.task_done()


A = ClosableQueue()
A.put(1)
A.put(2)
A.put(3)
A.close()

for item in A:
    print(item)  # expects 1, 2, 3