Facing issue while using sctp_recvmsg() and epoll for SCTP functionality implementation for Delphi application in Linux

Question

I am trying to implement SCTP support for an existing delphi application in Linux platform. I am using lksctp to support SCTP protocol. Also I am trying to use epoll(level triggered mode with EPOLLONESHOT flag) for event notification on the sockets. I am facing issue with the event handling and I am not getting where I am doing wrong or what I missed. Below is my scenario, It would be great if someone could help here to understand what is going wrong.

Currently from the server side I could see the server is reading(using sctp_recvmsg function) and processing same data repeatedly. Unfortunalely I could not post the code as the application code is very big.

Belows is the process flow of the application from the server side.

1. When the application starts epoll instance is created and the main thread creates a new thread which waits on the epoll handle for any network events(like EPOLLIN, EPOLLOUT etc..). The thread function waits on the epoll instance using epoll_wait() and once any event is notified by epoll_wait(), then the event type (like EPOLLIN, EPOLLOUT etc..) is recorded in a structure along with the socket descriptor and is pushed into a queue. Then the main application loop retrieves the structure one by one from the queue and performs the processing(sctp_send() or sctp_recvmsg() etc..) accordingly.

2. Down the line, the application creates a listening socket and adds(with EPOLLONESHOT flag enabled) this socket descriptor to the epoll handle created in step#1. Also the application starts listening on this listening socket.

3. Once the server receives a connection request from the client, the epoll handle signals the EPOLLIN event and the thread function pushes a structure with the event type and socket FD to the queue.

4. The main application loop retrieves the structure (with event details pushed to the queue in step#3) and processes the above connection request and it accepts the connection. At this point(ie after accepting connection) the listening socket is again re-armed for event notification using epol_ctl() and 'EPOLL_CTL_MOD'(This was done because of the EPOLLONESHOT used).

5. The newly accepted socket also now added(with EPOLLONESHOT flag) to the epoll handle created in step#1 for event monitoring.

6. Now the server receives a message ('UP' message) from the client on the newly accepted socket and epoll signals the EPOLLIN event.And the thread function pushes a structure with the event type and socket FD to the queue.

7. The main application loop retrieves the structure from the queue and does the processing, that means the Server now reads the 'UP' message from the newly accepted socket using sctp_recvmsg(). Immediately after this point the socket is re-armed for event notification using epol_ctl() and 'EPOLL_CTL_MOD'(This was done because of the EPOLLONESHOT used).

8. In response to the 'UP' message received in step#7 server sends 'UP_ACK' and 'NTFY' message back to client.

9. Client receives the 'UP_ACK' and 'NTFY' message from server and send back 'ACTIVE' message to server.

10. Now in the server side the epoll handle signals an EPOLLIN event and the server again tries to read the message using sctp_recvmsg(), now when the servers reads the message it is getting the same initial 'UP' message received in step#6 rather than the 'ACTIVE' message in step#9. Now the server agains sends 'UP_ACK' and 'NTFY' back to client and this goes on.

I checked from client side and i see it send the 'UP' message only one time.

1. Can someone give a hint on why the server is receiving the 'UP' message multiple times?
2. Once the server calls the first 'sctp_recvmsg()' for the 'UP' message shouldn't it remove the message from socket?.
3. Also can someone suggest the positions where I do the re-arming of sockets for event notification is correct? Or should I be doing it in the thread function where it is waiting for events using epoll_wait() ?