Why EPOLLOUT changes how EPOLLIN is handled?

Question

The documentation is unclear as to whether events are combined or not and my tests show that they are in some occasions but not always.

Consider man 7 epoll:

Since even with edge-triggered epoll, multiple events can be generated upon receipt of multiple chunks of data, the caller has the option to specify the EPOLLONESHOT flag...

and the Q&A section:

Q7 If more than one event occurs between epoll_wait(2) calls, are they combined or reported separately?

A7 They will be combined.

I assume the first statement from the manual means that you can receive more than one EPOLLIN event in situations like reading from a socket, a packet arrives, you read it, then another packet arrives. And the answer from the Q&A section is talking about different events like EPOLLIN and EPOLLOUT. Please correct me if I'm wrong.

I was playing around with some code in order to understand better how epoll works and it appears to behave differently regarding the same kind of event based on whether another one is set or not. More precisely, if I'm waiting only for EPOLLIN, multiple inputs generate a single event, but if I'm waiting for both EPOLLIN and EPOLLOUT, multiple inputs generate multiple events.

Here's the code I used to test this:

#include <stdio.h>
#include <fcntl.h>
#include <sys/epoll.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char* argv[])
{
  struct epoll_event ev = {EPOLLIN|EPOLLOUT|EPOLLET};
  int epoll = epoll_create1(0);
  epoll_ctl(epoll, EPOLL_CTL_ADD, 0, &ev);

  //async stdin
  int flags = fcntl(0, F_GETFL);
  flags |= O_NONBLOCK;
  fcntl(0, F_SETFL, flags);

  while(1){
    struct epoll_event events[64];
    int n = epoll_wait(epoll, events, 64, -1);

    printf("Event count: %d\n", n);

    if(events[0].events == EPOLLIN)
      printf("EPOLLIN only\n\n");

    else
    if(events[0].events == (EPOLLIN|EPOLLOUT))
      printf("EPOLLIN and EPOLLOUT\n\n");

    else
      printf("EPOLLOUT only\n\n");

    char buffer[256];
    read(0, buffer, 256);

    sleep(1);
  }
  return 0;
}

The output after pressing return shows that both EPOLLIN and EPOLLOUT are received, this message appears as many times as return was pressed, then it shows only EPOLLOUT being generated.

But if you compile the program without the EPOLLOUT flag and press return many times, a single event will be reported only once.

If I remove the read call, EPOLLIN continues to be reported when EPOLLOUT is set, but doesn't when only EPOLLIN is set.

Is the behavior dependent on which events it's waiting for or there's something wrong with my test code? If it is dependent, can I rest assured it won't change in the future?

Answer 1

I believe you are observing the effects of undefined behavior because you are abusing the API.

Specifically, you are passing STDIN_FILENO (i.e., 0) to epoll_ctl and asking to wait for EPOLLOUT on a file descriptor that is read only. What is likely happening is that the operating system is trying to tell you that there is something wrong with the write direction of the file descriptor.

Also, when using edge-triggered mode, you should continue your I/O until you it see EAGAIN. The epoll_wait call returns when the operation would no longer block.

I modified your program to use a socket instead, and read from the socket until EAGAIN, and it behaves as I would have expected.

In my version, I created a socket pair, and a thread that reads from STDIN_FILENO and writes to one of the pair of sockets. The main body loop then epoll_waits on the other socket.

When I start the program, it returns on the first call to epoll_wait to report writeable, but blocks on the next iteration:

Event count: 1
EPOLLOUT only

When I type input, it reports both readable and writeable, and then blocks on epoll_wait in the next iteration, as expected:

asdf
Event count: 1
EPOLLIN and EPOLLOUT

The code I used is below. First, the thread:

static void * iothread (void *svp) {
    int *sv = svp;
    char buf[256];
    ssize_t r;
again:
    while ((r = read(0, buf, sizeof(buf))) > 0) {
        ssize_t n = r;
        const char *p = buf;
        while (n > 0) {
            r = write(sv[1], p, n);
            if (r < 0) {
                if (errno == EINTR) continue;
                break;
            }
            n -= r;
            p += r;
        }
        if (n > 0) break;
    }
    if (r < 0 && errno == EINTR) {
        goto again;
    }
    close(sv[1]);
    return NULL;
}

Then, the main body:

int main(int argc, char* argv[]) {
  int sv[2];
  struct epoll_event ev = {EPOLLIN | EPOLLOUT | EPOLLET};
  int epoll = epoll_create1(0);
  pthread_t t;

  socketpair(AF_LOCAL, SOCK_STREAM, 0, sv);
  pthread_create(&t, NULL, iothread, sv);
  epoll_ctl(epoll, EPOLL_CTL_ADD, sv[0], &ev);
  while(1){
    struct epoll_event events[64];
    int n = epoll_wait(epoll, events, 64, -1);

    printf("Event count: %d\n", n);

    if(events[0].events == EPOLLIN)
      printf("EPOLLIN only\n\n");
    else
    if(events[0].events == (EPOLLIN|EPOLLOUT))
      printf("EPOLLIN and EPOLLOUT\n\n");
    else
      printf("EPOLLOUT only\n\n");

    char buffer[256];
    ssize_t r;
again:
    r = recv(sv[0], buffer, 256, MSG_DONTWAIT);
    if (r > 0) goto again;
    if (r < 0 && errno == EAGAIN) {
        sleep(1);
        continue;
    }
    break;
  }
  return 0;
}