CODEWITHSUNDEEP
c#arrays
greenoldman
greenoldman

Reputation: 21062

How to read binary array directly from the stream?

Here by "directly" I mean without temporary byte[] array.

The problem is, for example I have array of ints or doubles on the disk, so currently I create two arrays -- byte array and int array (in case of ints). The former is just for reading, the latter is the actual output.

Since Stream can read only to byte array I read it to the first array, than copy all the data to the second. It works, but it really hurts me (I am not talking about performance here).

So, how to read the array without temporary array? Using C# unsafe context is fine for me.

So far I tried two approaches: I looked if it is possible to create an array reusing allocated memory and second which looked more promising -- I could get pointer to the result/second array and in unsafe context I could cast it to byte* pointer. Considering my needs it is 100% safe and valid, however byte* pointer is not a byte[] array in C# world and I cannot find the way to cast pointer to array.

Code:

void ReadStuff(Stream stream, double[] data)
{  
  var dataBytes = new byte[data.Length * sizeof(double)];
  stream.Read(dataBytes, 0, dataBytes.Length);
  Buffer.BlockCopy(dataBytes, 0, data, 0, dataBytes.Length);
  // ...
}

Upvotes: 0

Views: 358

Answers (1)

JonasH
JonasH

Reputation: 36361

There is no way I know of to copy data directly from a stream to a typed array. But you can process the data in chunks, limiting your memory overhead to a fixed amount. The memory will be copied twice, but this is unavoidable as far as I know.

For example:

            public static void ReadArrayDataChunked(BinaryReader binaryReader, Array target, Type type, int bufferSize = 4096)
    {
        var buffer = new byte[bufferSize];
        var tSize = Marshal.SizeOf(type) ;

        var remainingBytes = target.Length * tSize;
        var targetPosition = 0;
        while (remainingBytes > 0)
        {
            var toRead = Math.Min(remainingBytes, buffer.Length);
            var bytesRead = binaryReader.Read(buffer, 0, toRead);
            Buffer.BlockCopy(buffer, 0, target, targetPosition, bytesRead);
            targetPosition += bytesRead;
            remainingBytes -= bytesRead;
        }
    }

Note that this only works for primitive types due to the BlockCopy, but this helps improve copy performance. You will need to read other types item by item.

Upvotes: 1

Related Questions