Managing a subprocess with gevent and stdin

Question

I have written a small program to start and stop a subprocess on command from read from stdin, using gevent to, hopefully, efficiently wait for both the command input, and for the process to finish.

It has one command R - run, which it read of stdin as a line.

R simply starts sh with the script sleep 2; echo "Hello, world!" if it is not already running

There are two greenlets, one reading command input and one waiting for a new process or waiting for the current process to finish. I use an gevent Event to toggle between the two.

My problem: The greenlet for the process is never allow to finish. The command greenlet appears to always be running and never relinquishes control. Why does the process greenlet never wake from its wait, even when the process is clearly finished?

The source is as follows:

import sys
from gevent import spawn, joinall, sleep
from gevent.subprocess import Popen
from gevent.fileobject import FileObjectPosix
from gevent.event import Event

process = None
trigger = Event()

def start_process():
    global process
    process = Popen(['/bin/sh', '-c', 'sleep 2; echo Hello, World!'])

def wait_process():
    global process
    while True:
        trigger.wait()
        print('INFO - Awaiting new process')
        trigger.clear()
        process.wait()
        process = None
        print('INFO - Process done')

def get_input():
    global process
    stdin_wrapped = FileObjectPosix(sys.stdin)
    while True:
        line = stdin_wrapped.readline().decode().strip()
        if line == 'R':
            if not process:
                start_process()
                trigger.set()
                print('OK - Running process')

            else:
                print('FAIL - Process already running')
        sleep(0)

def main():
    joinall([
        spawn(get_input),
        spawn(wait_process)
    ])

if __name__ == '__main__':
    main()

A session looks like this, with a >2s gap between the Rs:

R
OK - Running process
INFO - Awaiting new process
Hello, World!
R
FAIL - Process already running

I expect to see:

R
OK - Running process
INFO - Awaiting new process
Hello, World!
INFO - Process done
R
OK - Running process

My initial ideas are one of two things are going wrong:

• This is not the correct way to read a file with gevent
• the subprocess wait event is not being used correctly, and it never wakes up. I have not seen an example where it is used like this, but Popen objects can be used with gevent.wait so I assumed this was ok.
• If I break the debugger the stack trace shows that is is waiting for the read from stdin to finish, I expected it to have some select like behaviour, and when two greenlets were waiting on something it would execution would resume in the greenlet that finished first.

Answer 1

I have two solutions to my own problem. Either of these will make my example program function as expected.

Pipe for stdin

Open the subprocess with Popen(..., stdin=PIPE). It makes sense that gevent won't work without that since it must wait on something.

Use FileObjectThread

This seems to work regardless of how the subprocess is created, not sure why 😐