Confusion on how Python generator works

Question

Below is a copy from a Python book, the explanation on how generator works is not clear to me at all.

gen.send.py

def counter(start=0): 
    n = start 
    while True: 
        result = yield n # A 
        print(type(result), result) # B 
        if result == 'Q': 
            break 
        n += 1 

c = counter() 
print(next(c)) # C 
print(c.send('Wow!')) # D 
print(next(c)) # E 
print(c.send('Q')) # F 

And the output of the above is:

$ python gen.send.py 
0 
<class 'str'> Wow! 
1 
<class 'NoneType'> None 
2 
<class 'str'> Q 
Traceback (most recent call last): 
  File "gen.send.py", line 14, in <module> 
    print(c.send('Q')) # F 
StopIteration
Learning Python. . VitalBook file.

Explanation from book:

We start the generator execution with a call to next (#C). Within the generator, n is set to the same value of start. The while loop is entered, execution stops (#A) and n (0) is yielded back to the caller. 0 is printed on the console.

#Q1: At this point, n=1, right? because n+=1 should be executed since print(type(result), result) is executed.

We then call send (#D), execution resumes and result is set to 'Wow!' (still #A), then its type and value are printed on the console (#B). result is not 'Q', therefore n is incremented by 1 and execution goes back to the while condition, which, being True, evaluates to True (that wasn't hard to guess, right?). Another loop cycle begins, execution stops again (#A), and n (1) is yielded back to the caller. 1 is printed on the console.

Q2: 'Wow!' is sent to who? n, start, or result? and how? If n='Wow!' and what is the consequence of n+=1 then?

At this point, we call next (#E), execution is resumed again (#A), and because we are not sending anything to the generator explicitly, Python behaves exactly like functions that are not using the return statement: the yield n expression (#A) returns None.

Q3: Why None? whose value (start, n, result) exactly is suspended in this generator

result therefore is set to None, and its type and value are yet again printed on the console (#B). Execution continues, result is not 'Q' so n is incremented by 1, and we start another loop again. Execution stops again (#A) and n (2) is yielded back to the caller. 2 is printed on the console.

Q4: Why 2? Why not 4, or 5 because of n+=1 statement?

And now for the grand finale: we call send again (#F), but this time we pass in 'Q', therefore when execution is resumed, result is set to 'Q' (#A). Its type and value are printed on the console (#B), and then finally the if clause evaluates to True and the while loop is stopped by the break statement. The generator naturally terminates and this means a StopIteration exception is raised. You can see the print of its traceback on the last few lines printed on the console.

Thanks in advance.

Answer 1

One way to tackle this kind of learning problems is to visualize what is happening at each step. Let us start from the beginning:

def counter(start=0): 
    n = start 
    while True: 
        result = yield n # A 
        print(type(result), result) # B 
        if result == 'Q': 
            break 
        n += 1

Nothing interesting happens here. It is just a function definition.

c = counter()

Normally it should execute the function right? But since there is a yield keyword, it simply returns an object which can be used to execute the function. Read the previous sentence again! That is the first thing to understand about generators.

print(next(c)) # C

This is the way you execute the function using the object c. You don't invoke it with (), but instead do next(c). This is the very first time your instructions are executed and it happens till it finds the next yield statement. Since this is at A and the value of n is 0 at this moment, it just prints 0 and exits from the function - it would be better say the function pauses here. Remember it has not even reached n +=1! That answers your Q1.

print(c.send('Wow!')) # D

Now some more interesting stuff happens. The generator c, which had previously stopped at yield n, now just resumes and the next immediate instruction it has to perform is to result = in the result = yield n statement at A. It is given the value which you send in! So now result = 'Wow' has just happened.

Rest of the execution is normal. It again comes out of the function when it hits the next yield n. Now n is the n+1 because it was incremented in the while loop. I hope you can guess how the rest of the code behaves.

print(c.send('Q')) # F

Now this statement is somewhat different because it sends in a value that actually breaks the loop which in turn also stops any further yields in this case. Since the generator no longer finds any yield statements, it just throws a StopIteration exception and stops. If there was a yield outside of the while loop, it would have returned that and paused again.

Answer 2

Think of yield like a special return statement. When you get to the result = yield n line, first the right side is executed, returning n, which is 0. The difference from a return is that the function doesn't stop, it pauses, so the next time you call c.send(17) or next(c) it will resume from the yield, replacing it by the value you send (17) or None if you use the next(c) way. So when you call the first time next(c) it returns 0 and pauses, when you call c.send('Wow!') it resumes printing the type and the value you send from inside the generator, returning 1 and pausing, and it goes on.

Maybe if you add the letters to the print statements you can see easier where each output line comes from:

def counter(start=0):
    n = start
    while True:
        result = yield n # A
        print("B:", type(result), result) # B
        if result == 'Q':
            break
        n += 1

c = counter()
print("C:", next(c)) # C
print("D:", c.send('Wow!')) # D
print("E:", next(c)) # E
print("F:", c.send('Q')) # F

This would output:

$ python gen.send.py
C: 0
B: <class 'str'> Wow!
D: 1
B: <class 'NoneType'> None
E: 2
B: <class 'str'> Q
Traceback (most recent call last):
  File "gen.send.py", line 14, in <module>
    print("F:", c.send('Q')) # F
StopIteration

So answering your questions:

1. n = 0 yet as it paused after yielding n. The print is from the print(next(c)) # C line. n += 1 will get executed after you resume the generator with c.send('Wow!').
2. What you pass to the send method works as if it replaced the yield where the generator paused, in this case result = yield n -> result = 'Wow!', so it is passed to result.
3. When you do next(c) is equivalent to doing c.send(None), so it resumes the execution replacing the yield for None (result = yield n -> result = None).
4. You woke up the generator 4 times: the first next(c) didn't reach any n += 1; the c.send('Wow!') reached it once; the second next(c) reached it another time; and the c.send('Q') didn't reach it as the break statement was executing getting out of the while loop.

Answer 3

Q1: No, n is still 0 at this point. The generator stopped running at A, and the outside code has printed the first yielded value at C. The generator doesn't start running until send is called as part of line D.

Q2: The string "Wow!" becomes the value of the yield expression in line A, so it gets assigned to result in the generator. It, along with its type, gets printed out on line B, which is your second line of output. Then n gets incremented, and the loop starts over, with n (1) getting yielded as the return value from c.send. That gets printed on line D, for the third line of output.

Q3: You resume the generator on line E by calling next on it, which is equivalent to c.send(None). So result gets the value None (which is of type NoneType), and that gets printed in the generator code.

Q4: I'm not sure I understand what you're asking here. You seem to understand the execution flow. The code never prints more numbers because the generator has ended. It incremented n twice, but after it got the Q, it quit.

For what it's worth, you're very unlikely to ever need to write code quite like this example. It's very rare to mix next calls with send calls (except for the first next on a coroutine you're going to call send on the rest of the time).

Answer 4

Q1: Not sure what the question is here

Q2: 'Wow!' is sent to result.

Q3: result is None because execution was resumed with next(), so nothing was sent to the yield expression. As a result, the default value of None is sent instead.

Q4: Why would 4 or 5 be printed? n += 1 has only executed twice.