Most Efficient Way To Consolidate Multiple Byte Arrays Received Over Async Network?

Question

Basically, what I'm looking to do is find an effective means of consolidating large amounts of data that would be too large for a suitable buffer size.

For something like an instant messenger setting a fixed buffer size is a fine solution as most people accept that instant messages tend to have a limit.

However if I were to want to send a whole text document multiple pages long, you would not want to have to send it 2048 characters at a time. Or whatever you define as the limit.

I've represented my current solution in some pseudo code:

public class Pseudo
{
    public const int m_BufferSize = 255;

    public void SendAllData(Socket socket, byte[] data)
    {
        int byteCount = data.Length;

        if (byteCount <= m_BufferSize)
        {
            Socket.Send(data);
        }

        else if (byteCount > m_BufferSize)
        {
            int wholeSegments = Math.Floor(byteCount / m_BufferSize);
            int remainingBytes = byteCount % m_BufferSize;
            int bytesSent = 0;
            int currentSegment = 1;

            string id = Guid.NewGuid();

            byte[] tempData = data;

            //Send initial packet creating the data handler object.
            Socket.SendInitial(id);

            while (bytesSent < byteCount)
            {
                if (currentSegment <= wholeSegments)
                {
                    Socket.Send(tempData[m_BufferSize]);
                    tempData.CopyTo(tempData, m_BufferSize);
                    bytesSent += m_BufferSize;
                }

                else
                {
                    Socket.Send(tempData[remainingBytes]);
                    bytesSent += remainingBytes;
                }

                currentSegment++;
            }

            //Let The Server Know Send Is Over And To Consolidate;
            Socket.SendClose(id);

        }
    }

    internal class DataHandler
    {
        string m_Identity;
        List<byte[]> m_DataSegments = new List<byte[]>();

        static Dictionary<string, DataHandler> 
            m_HandlerPool = new Dictionary<string, DataHandler>();

        public DataHandler(string id)
        {
            m_Identity = id;

            if (!m_HandlerPool.ContainsKey(id))
            {
                m_HandlerPool.Add(this);
            }
        }

        public void AddDataSegment(byte[] segment)
        {
            m_DataSegments.Add(segment);
        }

        public byte[] Consolidate(string id)
        {
            var handler = m_HandlerPool(id);
            List<byte> temp = new List<byte[]>();

            for (int i = handler.m_DataSegments.Length; i >= 0; i--)
            {
                temp.Add(handler.m_DataSegments[i]);
            }

            Dispose();
            return temp;
        }

        void Dispose()
        {
            m_DataSegments = null;
            m_HandlerPool.Remove(this);
        }
    }
}

Basically what this is doing is assigning an identifier to individual packets so that they can be using AsyncEventArgs, as the may not necessarily all be received without being interrupted so I can't really rely on index.

These are then stored in the object 'DataHandler' and consolidated into a single byte array.

The problem is, as you can tell, it's going to add a lot of overhead in what I had hoped to be a high-performance socket server. Even if I were to pool the handler objects, the whole thing feels crufty.

Edit: It's also going to require a delimiter which I really don't want to use.

So, what would be the most efficient way of accomplishing this?

Edit: Example code for the method of processing data, this came from one of the async code projects.

    internal void ProcessData(SocketAsyncEventArgs args)
    {
        // Get the message received from the client.
        String received = this.sb.ToString();

        Console.WriteLine("Received: \"{0}\". The server has read {1} bytes.", received, received.Length);

        Byte[] sendBuffer = Encoding.ASCII.GetBytes(received);
        args.SetBuffer(sendBuffer, 0, sendBuffer.Length);

        // Clear StringBuffer, so it can receive more data from a keep-alive connection client.
        sb.Length = 0;
        this.currentIndex = 0;
    }

This is populating the user token data which is what is referenced by this.

                    // Set return buffer.
                    token.ProcessData(e);

Answer 1

Usually high performance server:

• use async send/receive methods
• don't store all received data in memory (if you want to store many data use some blob storage services)
• if you need, you can store some unordered pieces in some reusable context, and resend it to storage as soon as posible
• reuse buffers, don't allocate new buffer on each receive/send action
• etc...

Answer 2

I assume you're using TCP/IP - and if so, I don't understand the problem you're trying to fix. You can keep sending data as long as the connection is stable. Under the hood, TCP/IP will automatically create numbered packets for you, and ensure they arrive in the same order they were sent.

On the receiving end, you will have to read to a buffer of a certain size, but you can immediately write the received data to a MemoryStream (or FileStream if you intend to store the data on disk).