On which threads SocketAsyncEventArgs Completed is called and where to process messages

Question

I have a standard implementation for receiving UDP packets asynchronously using SocketAsyncEventArgs. What I do not understand from docs and some googling is if I should do the real work of processing messages inside the callback itself, like this comment indicates in the complete implementation I am referring to, or I should offload processing to other threads e.g. via ConcurrentQueue or BlockingCollection.

My concerns are the following:

• If processing directly in the callback, could it decrease receiving performance or introduce random delays due to temporary thread pool starvation or some other implementation detail?
• Due to slight delays in processing (vs. just offloading to a collection) could more packages be dropped due to a full buffer?
• Could more packages be reordered in the sense that the callback is called in different order than packages are actually arriving from a network.

So what is the best practice or intended way of processing messages using SocketAsyncEventArgs to ensure minimum missed datagrams, no additional reordering of callback calls and no additional delays?

And a related question - does ReceiveAsync guarantees any order at all or at least tries to call the callback in the same order that packages are received from a network, or I should use blocking receives for that? The target use case is to subscribe to 6-8 UDP channels, within each of them the order is highly important. Running a number of blocking threads looks more complex than just handling callbacks, but not that hard if only such solution guarantees message order.

Answer 1

what is the best practice or intended way of processing messages using SocketAsyncEventArgs to ensure minimum missed datagrams

Frankly, that's really a matter of opinion to some extent, as well as very dependent on your exact scenario. As a general rule though, your I/O completion routine should be fast. If your processing is fast, then it's fine to do that in the routine. If not, you should do as little work as possible, i.e. just to move the data to a queue where it can be handled elsewhere, and then return from the completion routine.

Keep in mind that "fast" here is relative. You just need to be faster than the network which, in spite of ever-increasing network speeds, is not very hard to do on a modern CPU. The network layer will buffer on your behalf, so assuming your workload throughput is greater than the throughput on the network, you are probably fine doing the work in the completion routine.

But really, it just depends. There's no way to say generally which is better. Each specific scenario is different.

does ReceiveAsync guarantees any order at all or at least tries to call the callback in the same order that packages are received from a network, or I should use blocking receives for that?

Blocking receives won't help.

The async methods all have the same characteristics: you can issue more than one at a time, and they will complete in the same order in which they were issued. But you still need to keep track of which order you issued the read operations. The buffers will be filled in the order you gave them to the network layer, but the completion routines may be executed out of order, because they are executed on threads, and the thread scheduler doesn't guarantee the order of thread execution. Just because one thread was made runnable before another, that doesn't mean it will actually get its next timeslice before the other one.

But it's actually worse than that:

The target use case is to subscribe to 6-8 UDP channels, within each of them the order is highly important.

If order is important in your scenario, you need to include sequence numbers in your datagrams and make sure you use those to put the data in the right order if those datagrams are received.

UDP doesn't guarantee ordering. Datagrams may be received in any order, independent of the order in which they are sent. UDP also doesn't guarantee delivery at all. Datagrams may be dropped at any time. UDP also doesn't guarantee uniqueness. A given datagram may be delivered multiple times.

If reliability and ordering is important in your scenario, you probably should be using TCP, and not UDP.

If the only thing you care about is ordering, then UDP may still work for you. And in that case, since you need sequence numbers in the datagrams anyway, it does make the "multiple concurrent read operations" scenario simpler, because the data itself comes with the sequence number, so you don't need to track that separately (e.g. in a state object associated with each read operation).