Implementing "SystemCalls" with Coroutines in Python

Question

Am currently reading through the tutorial doc http://www.dabeaz.com/coroutines/Coroutines.pdf and got stuck at the (pure coroutine) multitask part, in particular the System call section.

The part that got me confused is

class Task(object):
    taskid = 0
    def __init__(self,target):
        Task.taskid += 1
        self.tid = Task.taskid # Task ID
        self.target = target # Target coroutine
        self.sendval = None # Value to send

    def run(self):
        return self.target.send(self.sendval)


def foo():
    mytid = yield GetTid()
    for i in xrange(5):
        print "I'm foo", mytid
        yield


class SystemCall(object):
    def handle(self):
        pass

class Scheduler(object):
    def __init__(self):
        self.ready = Queue()
        self.taskmap = {}

    def new(self, target):
        newtask = Task(target)
        self.taskmap[newtask.tid] = newtask
        self.schedule(newtask)
        return newtask.tid

    def schedule(self, task):
        self.ready.put(task)

    def mainloop(self):
        while self.taskmap:
        task = self.ready.get()
        try:
            result = task.run()
            if isinstance(result,SystemCall):
                result.task = task
                result.sched = self
                result.handle()
                continue
            except StopIteration:
                self.exit(task)
                continue
        self.schedule(task)

And the actual calling

sched = Scheduler()
sched.new(foo())
sched.mainloop()

The part I don't understand is that how the tid got assigned to mytid in foo()? In the order of things, it seems to be like (starting from sched.mainloop()). Please correct me if I get the flow wrong.

Assumptions: let's name some of the things

the_coroutine = foo()
scheduler = Scheduler
the_task = scheduler.new(the_coroutine) # assume .new() returns the task instead of tid

1. scheduler: .mainloop() is called
2. scheduler: the_task.run() is called
3. the_task: the_coroutine.send(None) is called
4. the_corou: yield GetTid(), and it returns an instance of GetTid to scheduler before the None in 3 is sent to the yield statement in loop. (am i right?)
5. the_corou: (also at the same time?) myTid is assigned as the instance of GetTid()?
6. scheduler: result = theTask.run() the_task.run() <an instance of GetTid>
7. scheduler: result is indeed an instance of SystemCall
8. scheduler: result.handle() is called
9. GetTid instance: scheduler.schedule(the_task)
10. scheduler: current iteration is done, start a new one
11. scheduler: (assume no other task in queue) the_task.run() is called
12. scheduler: the_coroutine.send() is called
13. I am lost?

When it reaches step 12, apparently the loop is already started and is able to print the tid whenever the scheduler runs the respective task. However, when exactly was the value of tid assigned to mytid in foo()? I am sure I missed something in the flow, but where (or even completely wrong)?

Then I notice the part where the Task object calls .send(), it returns a value, so .send() returns a value?

Answer 1

You seem to have omitted the Scheduler's new method, which is almost certainly where the assignment occurs. I imagine the result of GetTid() that is yielded is immediately sent back into the coroutine using .send().

Regarding .send(), yes you're correct, .send() returns a value.

Try the example below to see what's going on. .send assigns a value to the variable on the left side of the = yield, and execution of code within the coroutine resumes until it hits the next yield. At that point, the coroutine yields, and whatever it yields is the return value of the .send.

>>> def C():
...     x = yield
...     yield x**2
...     print 'no more yields left'
...
>>> cor = C()
>>> cor.next()
>>> yielded = cor.send(10)
>>> print yielded
100
>>> cor.next()
no more yields left
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Moving some of the in-comment points into the answer

So, how does x = yield y work?

When the coroutine hits that line of code, it yields the value y, then halts execution, waiting for someone to invoke its .send() method with an argument.

When someone does invoke .send() whatever is the argument in the .send is assigned to variable x, and the coroutine begins to execute code from there upto the point of its next yield statement.

Edit: Whoa boy it gets even more complex... I skimmed through this David Beazley presentation before, but to be honest I'm better acquainted with his other two talks on generators... Going through the linked material, it looks like this definition of GetTid is what we're after.

class GetTid(SystemCall):
    def handle(self):
        self.task.sendval = self.task.tid
        self.sched.schedule(self.task)

I quote from his presentation: "The operation of this is little subtle". haha.

Now look at the mainloop:

if isinstance(result,SystemCall):
    result.task = task
    result.sched = self
    result.handle() # <- This bit!
    continue

result here is the GetTid object, which runs its handle method which sets its' task's sendval attribute to the task's tid, and the schedules the task by putting it back in the queue.

Once the task is retrieved from the queue, the task.run() method is run again. Let's look at the Task object definition:

class Task(object):
...
    def run(self):
        return self.target.send(self.sendval)

When task.run() is invoked for this second time, it will send its sendval value (which was previously set by result.handle() to its tid) to its .target - the foo coroutine. This is where the foo coroutine object finally receives the value of its mytid.

The foo coroutine object runs until its next yield, printing its message along the way, and returns None (because there's nothing to the right of yield). That None is the return value of the task.run() method.

That is NOT an instance of a SystemCall, so the task is not handled/scheduled upon the second pass.

Other evil things probably happen too but that's the flow that I'm seeing for now.