High CPU usage in System.net.Socket.beginReceive()

Question

I have a server software that has a single listening socket that then spans off multiple sockets (10 -30) which I then stream data to.

If I startup my application it used about 2-3% cpu usage on my 8 vCPU VM. After some time, generally 1-2 weeks the application suddenly starts using 60-70% cpu usage, and the thread count seems to stay stable, it does not increase.

I have run my C# profiler on my code and it comes down to the following line of code System.net.Socket.beginReceive().

I am using .net async sockets. below is my ReceiveCallBack My suspicion is that I am not handling the case when bytesRead is NOT >0. How should I modify my function below to handle that case correctly?

public static void ReadCallback(IAsyncResult ar)
        {
            SocketState tmpRef = null;
            try
            {
                if (ar != null)
                {
                    tmpRef = (SocketState)ar.AsyncState;
                    if (tmpRef != null)
                    {
                        // Read data from the client socket. 
                        int bytesRead = tmpRef.WorkSocket.Client.EndReceive(ar);
                        //Start Reading Again
                        tmpRef.BeginReading(tmpRef._receievCallbackAction);
                        if (bytesRead > 0)
                        {
                            // Check if we have a complete message yet
                            for (var i = 0; i < bytesRead; i++)
                            {
                                if (tmpRef._receiveBuffer[i] == 160)
                                {
                                    var tmpBuffer = new byte[i];
                                    Array.Copy(tmpRef._receiveBuffer, tmpBuffer, i);
                                    //Execute callback
                                    tmpRef._receievCallbackAction(tmpBuffer);
                                    break;
                                }
                            }
                        }
                    }
                }
            }
            catch (Exception e)
            {
                if (tmpRef != null)
                {
                    //Call callback with null value to indicate a failier
                    tmpRef._receievCallbackAction(null);
                }
            }
        }

Full code: (Sorry don't want to dirty the post) https://www.dropbox.com/s/yqjtz0r3ppgd11f/SocketState.cs?dl=0

Answer 1

You are explaining that the problems occurs after 1-2 weeks, which is quite rare then. I would suggest you to orientate your researchs by improving the exception handling in your readcallback.

Within this exception handling it turns out that you are invoking the callbackAction with null.

Maybe you should consider answering the following questions :

1. How does the callbackAction behaves when invoked with null tmpRef._receievCallbackAction(null);

2. What kind of exception is caught? If it is a SocketException, maybe look at the ErrorCode, which might give you an indication

3. Would it be possible to dump the stack trace to know exactly where it fails ?

Some other weak point : the begin receive uses this as state object. WorkSocket.Client.BeginReceive(_receiveBuffer, 0, BufferSize, 0, ReadCallback, this);

So it means that the thread safeness of the readcallback is not entirely guaranteed, because the call to BeginReading will occurs while you didn't process the _receiveBufferyet.

Answer 2

The problem is if you do not have enough bytes yet your code spins forever waiting for the next byte to show up.

What you need to do is make a messageBuffer that survive between calls and write to that till you have all the data you need. Also, by the way your code looks you look have the opportunity to overwrite tmpRef._receiveBuffer before you have copied all the data out, your BeginReading needs to start after the copy if you are sharing a buffer.

public class SocketState
{
    private readonly List<byte> _messageBuffer = new List<byte>(BufferSize);

    //...

    /// <summary>
    /// Async Receive Callback
    /// </summary>
    /// <param name="ar"></param>
    public static void ReadCallback(IAsyncResult ar)
    {
        SocketState tmpRef = null;
        try
        {
            if (ar != null)
            {
                tmpRef = (SocketState)ar.AsyncState;
                if (tmpRef != null)
                {
                    // Read data from the client socket. 
                    int bytesRead = tmpRef.WorkSocket.Client.EndReceive(ar);
                    if (bytesRead > 0)
                    {
                        //Loop over the bytes we received this read
                        for (var i = 0; i < bytesRead; i++)
                        {   
                            //Copy the bytes from the receive buffer to the message buffer.
                            tmpRef._messageBuffer.Add(tmpRef._receiveBuffer[i]);
                            // Check if we have a complete message yet
                            if (tmpRef._receiveBuffer[i] == 160)
                            {
                                //Copy the bytes to a tmpBuffer to be passed on to the callback.
                                var tmpBuffer = tmpRef._messageBuffer.ToArray();
                                //Execute callback
                                tmpRef._receievCallbackAction(tmpBuffer);
                                //reset the message buffer and keep reading the current bytes read
                                tmpRef._messageBuffer.Clear();
                            }
                        }

                        //Start Reading Again
                        tmpRef.BeginReading(tmpRef._receievCallbackAction);
                    }
                }
            }
        }
        catch (Exception e)
        {
            if (tmpRef != null)
            {
                //Call callback with null value to indicate a failier
                tmpRef._receievCallbackAction(null);
            }
        }
    }

//...