Can't do blocking read from named pipe (FIFO) in Linux

Question

It is very weird that I can't seem to make this work. This is my architecture: I have a named pipe which will communicate between a always-running root reader process and multiple application writer processes. The reader process has to be blocking while the writers are nonblocking. Therefore this is what I do in the reader process which will run with root privilege.

reader.c

#define PIPE_ID "/dev/shm/mypipe"
// This function configures named pipe
void configure_pipe() {
  // So that others can write
  umask(0000);
  if(mkfifo(PIPE_ID, S_IWUSR | S_IRUSR | S_IRGRP | S_IROTH | S_IWGRP
        | S_IWOTH) != 0) {
    perror("mkfifo error\n");
  }
}

In main function:

int main (int argc, char **argv)
{
  int Process_Pipe_ID;
  configure_pipe();

  // main loop which never ends
  while(1) {
    if((Process_Pipe_ID = open(PIPE_ID, O_RDONLY | O_NONBLOCK)) < 0) {
      perror("pipe file open error\n");
      exit(-1);
    }
    int bytes_read = 0;
    Message_Struct_t msg;

    // loop to keep reading from the pipe if there are messages to be read
    while((bytes_read = read(Process_Pipe_ID, &msg, sizeof(Message_Struct_t))) != 0) {
      if(bytes_read < 0) {
        perror("Pipe reading error in Scheduling agent\n");
        exit(-1);
      }
      printf("Read: %d, first: %d, second: %d\n", bytes_read, msg.first, msg.second);
      fflush(stdout);
    }
    close(Process_Pipe_ID);
  }
}

I expect this reader to be not blocked on open but it should keep reading from the named pipe if there's something on the pipe. Then, if it receives 0 which means EOF (nothing available in pipe) then it should close the file descriptor and open it again to keep trying reads from the pipe. It is kinda busy-waiting.

I expect the bytes_read to be exactly the sizeof(Message_Struct_t) (24 bytes) because I setup my writer as atomic. 24 bytes is less than PIPE_BUF, therefore Linux assures that it is atomic as long as I don't exceed the size-limit of a pipe. I am no way exceeding the size-limit. My writer programs are like clients; they come, execute and terminate. So the writing side of the pipe is not always open. This is my very simple writer:

writer.c

void writeInts(int first, int second) {
  Process_Pipe_ID = open(PIPE_ID, O_WRONLY | O_NONBLOCK);
  Message_Struct_t msg;
  msg.first = first;
  msg.second = second;
  int num;
  if((num = write(Process_Pipe_ID, &msg, sizeof(msg))) < 0) {
    perror("Error in writing\n");
    exit(-1);
  }  else
    printf("%d bytes wrote to pipe.\n", num);
  close(Process_Pipe_ID);
}

However, I am getting very weird output. I don't get anything on screen (for the reader.c) until I press enter. When I press enter I get the following:

Read: 1, first: 0, second: 0
Read: 1, first: 0, second: 0
Read: 1, first: 0, second: 0

When I press some other key and then enter, I get this:

Read: 1, first: 0, second: 0
aa
Read: 3, first: 0, second: 0
aaa
Read: 4, first: 0, second: 0

I have no idea what really is happening and how to go about this. I want to have a blocking read while the writers are non-blocking and atomic. I have searched a lot and then written the code but it is very weird that I can't get it to work.

Answer 1

It is very weird that I can't seem to make this work.

Well, not really. You have strange requirements and fragile assertions you try to go by.

The reader process has to be blocking.

Noooo... Why would you make such a limitation? Consider

ssize_t blocking_read(fd, void *buf, size_t len)
{
    struct timeval  timeout;
    fd_set          fds;
    int             r;
    ssize_t         n;

    /* First, do a speculative read, just in case
       there is data already available. */
    n = read(fd, buf, len);
    if (n >= 0)
        return n;
    else
    if (n != -1) {
        /* Paranoid check, will never happen .*/
        errno = EIO;
        return -1;
    } else
    if (errno != EAGAIN && errno != EWOULDBLOCK)
        return -1;

    /* Wait for data to become available. */
    FD_ZERO(&fds);
    while (1) {
        FD_SET(fd, &fds);
        timeout.tv_sec = 60; /* One minute */
        timeout.tv_usec = 0; /* and no millionths of seconds */

        r = select(fd + 1, &fds, NULL, NULL, NULL, &timeout);
        if (r < 0)
            return -1; /* errno set by select() */
        else
        if (!r)
            continue;  /* Timeout */

        n = read(fd, buf, len);
        if (n >= 0)
            return n;
        else
        if (n != -1) {
            /* Paranoid check, will never happen .*/
            errno = EIO;
            return -1;
        } else
        if (errno != EAGAIN && errno != EWOULDBLOCK)
            return -1;
    }
}

It acts like a blocking read on both blocking and nonblocking descriptors. It does wake up once per minute if nothing happens, but you can tune that until it is long enough to not matter. (I would consider values between one second and 86400 seconds, about one day. Any longer is just silliness. Remember it is a timeout, and not an ordinary sleep: any signal delivery or incoming data will immediately wake it up.)

With this, you can initially create the FIFO in 0400 mode (r--------), open it O_RDONLY | O_NONBLOCK in the reader, then use e.g. fchmod(fifofd, 0222) to change its mode (0222 = -w--w--w-) to allow writers. None of this blocks. None of the writer attempts to open the FIFO will succeed until the reader is ready.

The reader does not open and close the FIFO; it just keeps calling blocking_read().

If writers open the FIFO nonblocking write-only (O_WRONLY | O_NONBLOCKING), they will fail with errno = ENXIO if there is no reader, or with errno = EACCES if the reader is running but not ready yet. When there is a reader, the writes will succeed unless the reader cannot keep up. (When the reader has its buffer full, writers will get an error with errno = EAGAIN or errno = EWOULDBLOCK.)

Writers can easily do a nonblocking write, with a customizable timeout to wait until writing becomes possible; it is a very similar function to blocking_read() above.

I expect the bytes_read to be exactly the sizeof(Message_Struct_t) (24 bytes) because I setup my writer as atomic. 24 bytes is less than PIPE_BUF, therefore Linux assures that it is atomic as long as I don't exceed the size-limit of a pipe.

In optimal conditions, perhaps.

For example, if a nefarious user does e.g. echo 1 > your_pipe just when a writer is writing a message, you lose the message boundaries. The reader gets the two bytes (1 and newline) and the initial part of the message, the next read gets the last two bytes of that message and the initial part of the next message, as long as there are writers writing into the socket as fast or faster than the reader can read.

Because pipes and FIFOs never preserve message boundaries, your approach is extremely fragile. A better approach would be to use a datagram socket that does preserve message boundaries.

I am no way exceeding the size-limit. My writer programs are like clients; they come, execute and terminate. So the writing side of the pipe is not always open.

You could easily exceed the size limit.

There is just the one pipe buffer, and it can become full (and therefore nonblocking writes fail), if there are more writers than the reader can keep up. It is easy to cause that to happen: for example, if the reader does anything with the data, using two concurrent writers (like a Bash for ((;;)) ; do printf 'message' > fifo ; done) will fill the buffer, and cause any nonblocking writers to fail with errno = EAGAIN or errno = EWOULDBLOCK.

This is not just theoretical; it is easy to prove using Bash and mknod in practice.

---

I have a feeling OP is constructing a disaster waiting to happen with their current mix of requirements, especially using a pipe (or FIFO) for datagram transfer.

Personally, I would use an Unix domain datagram socket bound to a pathname, probably /var/run/yourservice. This would guarantee message boundaries (two different messages will not be mixed, like they can with pipes or FIFOs). Both the reader and writers could use ancillary data to pass an SCM_CREDENTIALS, which allows the reader to verify the user ID and group ID the writer used.

(The writer can choose between real or effective identities. The kernel always verifies the SCM_CREDENTIALS ancillary message fields, and will not allow incorrect data to be sent. In other words, the SCM_CREDENTIALS ancillary message fields will always be correct at the moment the message was sent.)

(Note that using a datagram protocol, the reader cannot verify the details of the process that sent the message, because by the time the reader receives the SCM_CREDENTIALS ancillary message, the original sender could have executed another process, or exited with the OS reusing the process ID for some other new process. To verify which executable was used to send a message, one would need a connection-oriented protocol, like Unix domain stream socket, with the writer sending two or three messages, all with the same SCM_CREDENTIALS ancillary message. This is quite tricky to do correctly, so most programmers consider such verification icky.)