CODEWITHSUNDEEP
c#memory-management
murmansk
murmansk

Reputation: 845

MemoryStream exhausting

I am writing some bytes into a MemoryStream. I have given it a size of 100 MB

  using (MemoryStream Stream = new MemoryStream())
        {             
                for ( loop 1400 time)
                {                        
                  func(i, OStream, Stream, );

                }

                lock (this)
                {                       
                 Stream.WriteTo(OStream);
                }

}

func(i, OStream, Stream)
{
  loop 19 times
      get buffer;
   Stream.Write(buffer, 0, buffer.Length);
}

Buffer is a byte[] with size varying from 65536 to 116454. I loop 1000 times. I am writing it in for loop, but i am getting OutOfMemoryException exception.

Is there any way expanding MemoryStream and avoid the exception?

Upvotes: 0

Views: 902

Answers (1)

Thomas Weller
Thomas Weller

Reputation: 59650

Considering the amout of data, that's minimum 6000*65536 bytes, which is <400 MB for the byte arrays and maximum 6000*116454 bytes, which is <700 MB. The MemoryStream will double its size whenever necessary, so it first has 100 MB, then 200 MB, then 400 MB, perhaps 800 MB.

Even if the garbage collector cannot clean up in between, you might think that this is a maximum total of 2200 MB (700 for byte arrays, 100 for the initial memory stream, 200 for the first expansion, ...). In fact, that may be the case and "enough" memory is left. However, it may not be available in a contiguous block.

What you're likely facing is a memory fragmentation issue. A large number of your byte arrays tends to exceed 85001 bytes, so they are allocated on the large object heap. The large object heap does not get compacted and thus suffers from fragmentation.

You can try to use the answer at How to use WinDbg to track down .NET OutOfMemoryExceptions. I also posted a memory fragmentation example in case you want to compare your situation to it.

Potential fixes

A quick fix would be to compile as x64, because you have way more memory available there. A cleaner approach depends on your real situation and requires an optimization of the algorithm, e.g. you could calculate the total size of the needed result in advance, so that the MemoryStream does not need to double its size.

It might also be good to use a single 200 kB byte array and re-fill it with temporary data instead of creating new arrays all over the place.

If your final task is to write the data into a file, use a FileStream and write it directly into a file.

You might also get rid of the MemoryStream completely and just keep all byte arrays together in a List<byte[]>.

And probably many other solutions...

Upvotes: 5

Related Questions