CODEWITHSUNDEEP
pythonlinuxasynchronousselect
tobib
tobib

Reputation: 2434

linux: how to wait for a process and file at the same time?

I am trying to build an event loop in python, mostly for educational purposes. I am not interested in solutions involving asyncio or similar because my goal is really to learn about the low-level linux APIs.

For file-like objects (pipes, sockets, …) there are the select/poll/epoll system calls. In python, these are wrapped in the selectors module. This is a simple loop that can wait for several files at the same time:

file1 = open('/var/run/foo.sock')
file2 = open('/var/run/bar.sock')

with selectors.DefaultSelector() as selector:
    selector.register(file1, selectors.EVENT_READ)
    selector.register(file2, selectors.EVENT_READ)

    while True:
        for key, mask in selector.select():
            if key.fileobj == file1:
                handle_data(file1)
            else:
                handle_data(file2)

For proceeses there is the wait system call. In python this is wrapped in the subprocess module. This is the code to wait for a single process to terminate:

proc = subprocess.Popen(['/usr/bin/foo'])
proc.wait()
handle_data(proc)

How can I mix those two, or even wait for more than one process at a time.

Both selector.select() and proc.wait() will take a timeout, so we can turn this into a semi-busy loop:

file1 = open('/var/run/foo.sock')
file2 = open('/var/run/bar.sock')
proc = subprocess.Popen(['/usr/bin/foo'])
timeout = 1

with selectors.DefaultSelector() as selector:
    selector.register(file1, selectors.EVENT_READ)
    selector.register(file2, selectors.EVENT_READ)

    while True:
        for key, mask in selector.select(timeout):
            if key.fileobj == file1:
                handle_data(file1)
            else:
                handle_data(file2)

        if proc:
            try:
                proc.wait(timeout)
                handle_data(proc)
                proc = None
            except TimeoutExpired:
                pass

I guess there must be a better way. How is this usually done?

Upvotes: 0

Views: 130

Answers (1)

tobib
tobib

Reputation: 2434

I found the relevant information when looking at the different implementations of child watchers in python's asyncio:

  • You can call os.waitpid() in a separate thread and write to a pipe when it is done. The other end of that pipe can be added to the select loop. This approach seems to be the most common.
  • You can write to a pipe when you receive SIGCHLD and again add the other end of that pipe to the select loop. Then you can call process.poll() to check which of the processes has terminated in O(n) time.
  • open_pidfd() is exactly what I was looking for. It is the cleanest solution, but only available on Linux >= 5.3. This is probably the way to go if you do not have to care about portability.

Upvotes: 1

Related Questions