CODEWITHSUNDEEP
c#streambinaryreader
user3428422
user3428422

Reputation: 4560

Byte[stream.length] - out of memory exception, best way to solve?

i am trying to read the stream of bytes from a file. However when I try to read the bytes I get a

The function evaluation was disabled because of an out of memory exception

Quite straightforward. However, what is the best way of getting around this problem? Is it too loop around the length at 1028 at a time? Or is there a better way?

The C# I am using 

BinaryReader br = new BinaryReader(stream fs);

// The length is around 600000000
long Length = fs.Length;

// Error here
bytes = new byte[Length];

for (int i = 0; i < Length; i++)
{
   bytes [i] = br.ReadByte();
}

Thanks

Upvotes: 0

Views: 9701

Answers (3)

Florian
Florian

Reputation: 5994

Well. First of all. Imagine a file with the size of e.g. 2GB. Your code would allocate 2GB of memory. Just read the part of the file you really need instead of the whole file at once. Secondly: Don't do something like this:

for (int i = 0; i < Length; i++)
{
   bytes [i] = br.ReadByte();
}

It is quite inefficient. To read the raw bytes of a stream you should use something like this: 

using(var stream = File.OpenRead(filename))
{
    int bytesToRead = 1234;
    byte[] buffer = new byte[bytesToRead];

    int read = stream.Read(buffer, 0, buffer.Length);

    //do something with the read data ... e.g.:
    for(int i = 0; i < read; i++)
    {
        //...
    }
}

Upvotes: 4

Dario
Dario

Reputation: 11

I believe that the instantiation of the stream object already reads the file (into a cache). Then your loop copies the bytes in memory to another array.

So, why not to use the data into "br" instead of making a further copy?

Upvotes: 0

MRebai
MRebai

Reputation: 5474

When you try to allocate an array, the CLR lays it out contiguously in virtual memory given to it by the OS. Virtual memory can be fragmented, however, so a contiguous 1 GB block may not be available, hence the OutOfMemoryException. It doesn't matter how much physical RAM your machine has, and this problem is not limited to managed code (try allocating a huge array in native C and you'll find similar results).

Instead of allocating a huge array, I recommend using several smaller arrays, an ArrayList or List, that way the Framework can allocate the data in chunks.

Hope that helps

Upvotes: 0

Related Questions