CODEWITHSUNDEEP
pythonlinuxpipe
Giovanni Mascellani
Giovanni Mascellani

Reputation: 1278

Linux FIFO not returning EOF when I expect it to

Let us consider the following Python code, to be executed by cpython on a Linux system (warning: it will try to create or overwrite files in /tmp/first, /tmp/second and /tmp/third).

#!/usr/bin/env python2
# -*- coding: utf-8 -*-

import subprocess
import os
import sys
import threading

class ThreadizedPopen(threading.Thread):

    def __init__(self, command, stdin_name, stdout_name):
        super(ThreadizedPopen, self).__init__()
        self.command = command
        self.stdin_name = stdin_name
        self.stdout_name = stdout_name
        self.returncode = None

    def run(self):
        with open(self.stdin_name, 'rb') as fin:
            with open(self.stdout_name, 'wb') as fout:
                popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None)
                popen.communicate()
                self.returncode = popen.returncode

def main():
    os.system('mkfifo /tmp/first')
    os.system('mkfifo /tmp/second')
    os.system('mkfifo /tmp/third')

    popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
    popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
    popen1.start()
    popen2.start()
    with open('/tmp/third') as fin:
        print fin.read()
    popen1.join()
    popen2.join()

if __name__ == '__main__':
    main()

I execute it then, on another shell, I write something in /tmp/first (say with echo test > /tmp/first). I would expect the Python program to quickly exit and print the same thing I fed to the first FIFO.

In theory it should happen that the string I wrote in /tmp/first gets copied over by the two cat processes spawned by my program to the other two FIFOs and then picked up by the main Python program to be wrote on its stdout. As soon as every cat process finished, it should close its end of the writing FIFO, making the corresponding reading end return EOF and triggering the termination of the following cat process. Looking at the program with strace reveals that the test string is copied correctly through all the three FIFOs and is read by the main Python program. The first FIFO is also correctly closed (and the first cat process exits, together with its manager Python thread). However the second cat process is stuck in a read() call, expecting data from its reading FIFO.

I do not understand why this happens. From the pipe(t) man page (which, I understand, covers also this kind of FIFOs) it seems that a read on a FIFO is returned EOF as soon as the writing end (and all its duplicates) are closed. According to strace this appears to be the trace (in particular, the cat process is dead, thus all its file descriptors are closed; its managing thread has closed its descriptors as well, I can see it in the strace output).

Can you suggest me why that happens? I can post the strace output if it can be useful.

Upvotes: 2

Views: 356

Answers (1)

jDo
jDo

Reputation: 4010

I found this question and simply added close_fds=True to your subprocess call. Your code now reads:

#!/usr/bin/env python2
# -*- coding: utf-8 -*-

import subprocess
import os
import sys
import threading

class ThreadizedPopen(threading.Thread):

    def __init__(self, command, stdin_name, stdout_name):
        super(ThreadizedPopen, self).__init__()
        self.command = command
        self.stdin_name = stdin_name
        self.stdout_name = stdout_name
        self.returncode = None

    def run(self):
        with open(self.stdin_name, 'rb') as fin:
            with open(self.stdout_name, 'wb') as fout:
                popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None, close_fds=True)
                popen.communicate()
                self.returncode = popen.returncode

def main():
    os.system('mkfifo /tmp/first')
    os.system('mkfifo /tmp/second')
    os.system('mkfifo /tmp/third')

    popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
    popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
    popen1.start()
    popen2.start()
    with open('/tmp/third') as fin:
        print fin.read()
    popen1.join()
    popen2.join()

if __name__ == '__main__':
    main()

I placed your code in a script called fifo_issue.py and ran it in a terminal. The script was idling as you'd expect (ignore mkfifo: cannot create fifo):

$ python fifo_issue.py 
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists

Then, in a second terminal, I typed:

$ echo "I was echoed to /tmp/first!" > /tmp/first

Back to the first terminal that's still running your idling threads:

$ python fifo_issue.py 
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists
I was echoed to /tmp/first!

After which python exited correctly

Upvotes: 1

Related Questions