Socket buffers the data it receives

Question

I have a client .NET application and a server .NET application, connected through sockets.

The client sends a string of 20 or so characters every 500 milliseconds.

On my local development machine, this works perfectly, but once the client and the server are on two different servers, the server is not receiving the string immediately when it's sent. The client still sends perfectly, I've confirmed this with Wireshark. I have also confirmed that the the server does receive the strings every 500 milliseconds.

The problem is that my server application that is waiting for the message only actually receives the message every 20 seconds or so - and then it receives all the content from those 20 seconds.

I use asynchronous sockets and for some reason the callback is just not invoked more than once every 20 seconds.

In AcceptCallback it establishes the connection and call BeginReceive

handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0, new AsyncCallback(ReadCallback), state);

This works on my local machine, but on my production server the ReadCallback doesn't happen immediately.

The BufferSize is set to 1024. I also tried setting it to 10. It makes a difference in how much data it will read from the socket at one time once the ReadCallback is invoked, but that's not really the problem here. Once it invokes ReadCallback, the rest works fine.

I'm using Microsofts Asynchronous Server Socket Example so you can see there what my ReadCallback method looks like.

How can I get the BeginReceive callback immediately when data arrives at the server?

--

UPDATE

This has been solved. It was because the server had a a single processor and single core. After adding another core, the problem was instantly solved. ReadCallback is now called immediately when the call goes through to the server.

Thankyou all for your suggestions!!

Answer 1

By the request of OP, duplicating my "comment/answer" here.

My guess was, the problem appeared because of thread scheduling on a single-core machine. This is an old problem, almost extinct in the modern age of hyper-threading/multi-core processors. When a thread is spawned in the course of execution of the program, it needs scheduled time to run.

On a single-core machine, if one thread continues to execute without explicitly passing control to OS scheduler (by waiting for mutex/signal or by calling Sleep), the execution of any other thread (in the same process and with lower priority) may be postponed indefinitely by the scheduler. Hence, in the case described, the asynchronous network thread was (most likely) just starved for execution time - getting only pieces from time to time.

Adding second CPU/core, obviously, fixed that by providing a parallel scheduling environment.

Answer 2

One approach might be to adjust the SO_SNDBUF option for the send side. SInce you are not running into this problem when both server/client are on the same box, it is possible that having a small buffer is throttling the send side since due to (a possible) slower sending rate between the servers. If the sender cannot send fast enough, then the send-side buffer might be filling up sooner.

Update: we did some debugging and turns out that the issue is with the application being slower.

Answer 3

It might be that the Nagle algorithm is waiting on the sender side for more packets. If you are sending small chunks of data, they will be merged in one so you don't pay a huge TCP header overhead for small data. You can disable it using: StreamSocketControl.NoDelay See: http://msdn.microsoft.com/en-us/library/windows/apps/windows.networking.sockets.streamsocketcontrol.nodelay

The Nagle algorithm might be disabled for loopback and this is a possible explanation of why it works when you have both the sender and the receiver on the same machine.