Converting non-seekable Stream to byte[] gives ArgumentOutOfRangeException

Question

I'm trying to GetResponseStream from a GET request and convert this into byte[]. The Stream is non-seekable so I can't access stream.Length. response.ContentLength is unreliable.

private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
    {
        byte[] buffer = new byte[bufferSize];
        int read = 0;
        int pos = 0;
        List<byte> bytes = new List<byte>();
        while (true) { // `bufferSize > read` does **not** mean stream end
            while (bufferSize == (read = stream.Read(buffer, pos, bufferSize))) {
                pos += read;
                bytes.AddRange(buffer);
            }
            if (read > 0) {
                byte[] _buffer = new byte[read];
                Array.Copy(buffer, _buffer, read);
                bytes.AddRange(_buffer);
            } else break;
        }
        return bytes.ToArray();
    }

An ArgumentOutOfRangeException gets thrown on the Read in the second iteration of the while loop. MSDN says

ArgumentOutOfRangeException offset or count is negative.

I know this can't be true because offset (pos) is 0 + read >= 0 and count (bufferSize) is 4096 so why am I getting exceptions thrown at me?

I'm trying to keep streamToBytes as generic as possible so I can use it in future async methods, too.

---

If how the request is made helps, here are the relevant bits

HttpWebRequest request = (HttpWebRequest)WebRequest.Create((new Uri("http://google.com")).ToString());
request.Method = "GET";
request.KeepAlive = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
byte[] responseBytes = streamToBytes(stream);

Answer 1

As a simpler alternative:

using(var memStream = new MemoryStream())
{
    stream.CopyTo(memStream);
    return memStream.ToArray();
}

---

Your code has two mistakes:

1. The offset on Read is an offset into the buffer, not the position in the stream (as @Ulugbek already noted). This means you don't need the pos variable anymore.
2. You cannot assume that bufferSize == read even before the stream reached its end. You need to check for > 0 instead.

So your reading loop becomes:

while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
{
    bytes.AddRange(buffer.Take(read));
}

You can now drop the special handling for the last block. Simplifying your code to:

private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
    byte[] buffer = new byte[bufferSize];
    int read = 0;
    List<byte> bytes = new List<byte>();
    while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
    {
        bytes.AddRange(buffer.Take(read));
    }
    return bytes.ToArray();
}

Using List<byte> instead of MemoryStream isn't a great idea either. AddRange need to iterate over the bytes individually, instead of using a low level copy operation. Replacing the list with a memory stream, the code becomes:

private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
    byte[] buffer = new byte[bufferSize];
    int read = 0;
    using(var bytes = new MemoryStream())
    {
        while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
        {
            bytes.Write(buffer, 0, read);
        }
        return bytes.ToArray();
    }
}

You could even split it into two parts, one does the copying into the memory stream, the other handles the creating of the memory stream and turning it into a byte array:

private static void CopyStream(Stream source, Stream destination, int bufferSize = 4096)
{
    byte[] buffer = new byte[bufferSize];
    int read = 0;
    while ((read = source.Read(buffer, 0, bufferSize)) > 0)
    {
        destination.Write(buffer, 0, read);
    }
}

private static byte[] StreamToBytes(Stream stream, int bufferSize = 4096)
{
    using(var memStream = new MemoryStream())
    {
        CopyStream(stream, memStream);
        return memStream.ToArray();
    }
}

This is probably very similar to what Stream.CopyTo does internally.

Answer 2

Change

stream.Read(buffer, pos, bufferSize)

to

stream.Read(buffer, 0, bufferSize)