CODEWITHSUNDEEP
linuxsockets
charfeddine.ahmed
charfeddine.ahmed

Reputation: 546

Safe way to handle closure of sockets managed by epoll

Using epoll_wait to manage multiple connections using multiple threads, there is a risk trying to release custom data associated with a closed socket.

Consider the following scenario, where T is the custom data :

  1. Data is received,
  2. Because of 1, thread A deblocks from epoll_wait and processes the event (access T)
  3. At same time, another thread B, wants to close the connection

Thread B can't assume that T can be safely deleted, eventhough the call to close will immediatly remove the socket from the epoll.

I had the following standard idea :

Maintain a variable within T that gets incremented each time a call to write/read returns EAGAIN, and gets decremented each time the socket is ready. When close is called, wait for that variable to go down to zero before deleting T. The issue I experienced is that if close is called, epoll_wait does not return an indication of a cancellation of previous calls to arm the socket.

Anybody had this same problem ? How did you managed to overcome it ?

Upvotes: 4

Views: 3035

Answers (2)

charfeddine.ahmed
charfeddine.ahmed

Reputation: 546

After many research, I found this recent and remarkable article : http://lwn.net/Articles/520012/ Basically it acknowledge the issue I am describing and speaks about a possible future patch to Linux kernel that allows to extend the epoll API in a way that solves the issue.

The extension bring a new command called : EPOLL_CTL_DISABLE. When it is issued, and by means of return value, the calling thread will know if some other thread has just been deblocked from epoll_wait upon same socket. This can help know the safe moment of closure and release of custom data.

Upvotes: 1

Nikolai Fetissov
Nikolai Fetissov

Reputation: 84159

At least three possible ways here:

  • Do not use threads, simple and clean, and usually works.

  • Have a dedicated thread do all file descriptor polling and publish events to a pool of worker threads that do actual I/O and processing.

  • Have one epoll(7) instance per thread, so threads manage non-intersecting sets of descriptors, with the exception of maybe the listening socket(s) to get these sets populated, and some control mechanism like eventfd(2), or self-pipe(2) to be able to shutdown the whole rig cleanly.

Hope this helps.

Upvotes: 4

Related Questions