C++/Python Processes IPC with FIFO: Can't read twice in a row from Python

Question

I'm having issues trying to make a C++ process communicate with a Python one through FIFO.

These processes use a 2 way communication through 2 FIFOs:

• the C++ writes on pythonread_fifo and viceversa the Python reads from it
• the Python process writes on cppread_fifo and viceversa the C++ reads from there

The idea would be that both processes write and read at every cycle. The Python one writes "woof" and the C++ writes "quack".

If one of the 2 processes is faster than the other (like the C++ executes 3 cycles while the python is still stuck at the first one) then I expect the other one to not suffer data loss (in my example, the python would receive quackquackquack).

My code seemed to work until I tried to let the Python process cycle twice in a row while the C++ was frozen, which triggered the error:

BlockingIOError: [Errno 11] Resource temporarily unavailable in line 24: os.read(fd[0], 20)

Here's the code; please note that only the Python process calls mkfifo because that's the desired behaviour in the project:

pipe.cpp

#define MAX_BUF 1024

const char * write_path = "python_cpp_comms/pythonread_fifo";
const char * read_path =  "python_cpp_comms/cppread_fifo";

int main(int argc, char**argv)
{
    int fd[2];
    char buf[MAX_BUF];

    printf("Opened\n");
    fd[1] = open(write_path, O_RDWR);
    fd[0] = open(read_path, O_RDONLY | O_NONBLOCK);
    int cycle = 1;
    for(;;) {
        printf("Cycle: %d\n",cycle++);
        int out = write(fd[1], "quack", 5);
        printf("Written %d bytes errno=%d:\n", out, errno);
        int in = read(fd[0], buf, MAX_BUF);
        printf("Received %d bytes errno=%d: %*.*s\n", in, errno, in,in, buf);
        std::cout << "Doing something \n";
        getchar();
    }
}

pipe.py

read_path =  "python_cpp_comms/pythonread_fifo"
write_path = "python_cpp_comms/cppread_fifo"

if __name__ == "__main__":
    try:
        os.mkfifo(read_path)
    except:
        pass
    try:
        os.mkfifo(write_path)
    except:
        pass

    print("Opening FIFO...")

    fd = []
    fd.append(os.open(read_path, os.O_RDONLY | os.O_NONBLOCK))
    fd.append(os.open(write_path, os.O_RDWR))
    cycle = 1

    while True:
        print("Cycle: {}".format(cycle))
        data = os.read(fd[0], 20)
        if len(data) == 0:
            print("Writer closed")
        print('Received: "{0}"'.format(data))
        print("writing 'woof'")
        os.write(fd[1], b'woof')
        cycle += 1
        print("Doing something\n")
        input()

As you can see, I'm using getchar/input in order to control the flow here.

So in a simple situation where I make the 2 processes move forward one by one (one cycle on C++ and one on Python and so on) the result is the following:

If I run 2 or more C++ cycles and then run a cycle on the Python process, no data is lost and it works just fine

If I run the Python process twice instead, WHILE THE C++ PROCESS IS RUNNING, then the Python process crashes. However, if the C++ process is CLOSED, the crash doesn't occur and the fifo works just fine

Answer 1

1. FIFO Blocking Behavior: By default, FIFO is blocking. This means a read operation in Python will wait until there’s data to read from the pipe. If the C++ process has already written data and exited (or closed the pipe), the Python side might hang waiting for more data. Make sure that the writing side (C++) remains open or is writing new data each time.

2. Opening the FIFO Properly:

   • In Python, open the FIFO in read mode:

     fifo = open("path_to_fifo", "r")
     

   • If the C++ side closes the FIFO after each write, Python’s read() may reach an EOF and stop waiting for additional data.
   • If the FIFO closes, you need to reopen it before each read to reset it in Python.

3. Use os.open for Fine Control: In Python, os.open allows more control over FIFO files:

   import os
   fifo = os.open("path_to_fifo", os.O_RDONLY | os.O_NONBLOCK)
   

4. C++ Code:

   • If C++ writes to the FIFO and closes it, make sure Python is aware that it needs to reopen the pipe for a subsequent read.
   • If C++ keeps the FIFO open, consider flushing the write buffer after each write:

     std::ofstream fifo("path_to_fifo");
     fifo << "your data" << std::endl;
     fifo.flush();  // Ensure data is immediately sent
     

5. Looping Reads in Python: If you want continuous reading:

   while True:
       with open("path_to_fifo", "r") as fifo:
           for line in fifo:
               print(line.strip())  # Process line