CODEWITHSUNDEEP
c#winformsout-of-memory
Razort4x
Razort4x

Reputation: 3406

why do I get System.OutOfMemoryException even when there is about 700Mb of RAM free?

I was running some test, to see the how my logging would perform is instead of doing File.AppendAllText I would first write to a memory stream and then copy to file. So, just to see how fast memory operation is I did this..

private void button1_Click(object sender, EventArgs e)
    {
        using (var memFile = new System.IO.MemoryStream())
        {
            using (var bw = new System.IO.BinaryWriter(memFile))
            {
                for (int i = 0; i < Int32.MaxValue; i++)
                {
                    bw.Write(i.ToString() + Environment.NewLine);
                }
                bw.Flush();
            }
            memFile.CopyTo(new System.IO.FileStream(System.IO.Path.Combine("C", "memWriteWithBinaryTest.log"), System.IO.FileMode.OpenOrCreate));
        }
    }

When i reached 25413324 I got a Exception of type 'System.OutOfMemoryException' was thrown. even though my Process Explorer says I have about 700Mb of free ram???

Here are the screen shots (just in case)

Process Explorer enter image description here

Here's the winform

enter image description here

EDIT : For the sake of more objects being created on heap, I rewrote the bw.write to this

bw.Write(i);

Upvotes: 6

Views: 7993

Answers (3)

leppie
leppie

Reputation: 117220

You should not be using a BinaryWriter to write text to a file. Use a TextWriter instead.

Now you are using:

for (int i = 0; i < Int32.MaxValue; i++)

This will write at least 3 bytes per write (number representation and newline). Times that by Int32.MaxValue and you need at least 6GB of memory already seeing that you are writing it to a MemoryStream.

Looking further at your code, you are going to write the MemoryStream to a file in any way. So you can simply do the following:

for (int i = 0; i < int.MaxValue; i++)
{
  File.AppendAllText("filename.log", i.ToString() + Environment.Newline);
}

or write to an open TextWriter:

TextWriter writer = File.AppendText("filename.log");

for (int i = 0; i < int.MaxValue; i++)
{
  writer.WriteLine(i);
}

If you want some memory buffer, which IMO is a bad idea for logging as you will lose the last bit of the writes during a crash, you can using the following the create the TextWriter:

StreamWriter(string path, bool append, Encoding encoding, int bufferSize)

and pass a 'biggish' number for bufferSize. The default is 1024.

To answer the question, you get an out of memory exception due to the MemoryStream resizing and at some point it gets to big to fit into memory (which was discussed in another answer).

Upvotes: 0

Roman Starkov
Roman Starkov

Reputation: 61382

First of all, you run out of memory because you accumulate data in the MemoryStream, instead of writing it directly to the FileStream. Use the FileStream directly and you won't need much RAM at all (but you will have to keep the file open).

The amount of physical memory unused is not directly relevant to this exception, as strange as that might sound.

What matters is:

  • that you have a contiguous chunk of memory available in the process' virtual address space
  • that the system commit does not exceed the total RAM size + page file size

When you ask the Windows memory manager to allocate you some RAM, it needs to check not how much is available, but how much it has promised to make available to every other process. Such promising is done through commits. To commit some memory means that the memory manager offered you a guarantee that it will be available when you finally make use of it.

So, it can be that the physical RAM is completely used up, but your allocation request still succeeds. Why? Because there is lots of space available in the page file. When you actually start using the RAM you got through such an allocation, the memory manager will just simply page out something else. So 0 physical RAM != allocations will fail.

The opposite can happen too; an allocation can fail despite having some unused physical RAM. Your process sees memory through the so-called virtual address space. When your process reads memory at address 0x12340000, that's a virtual address. It might map to RAM at 0x78650000, or at 0x000000AB12340000 (running a 32-bit process on a 64-bit OS), it might point to something that only exists in the page file, or it might not even point at anything at all.

When you want to allocate a block of memory with contiguous addresses, it's in this virtual address space that the RAM needs to be contiguous. For a 32-bit process, you only get 2GB or 3GB of usable address space, so it's not too hard to use it up in such a way that no contiguous chunk of a sufficient size exists, despite there being both free physical RAM and enough total unused virtual address space.

Upvotes: 9

Matthew Watson
Matthew Watson

Reputation: 109537

This can be caused by memory fragmentation.

Large objects go onto the large object heap and they don't get moved around to make room for things. This can cause fragmentation where you have gaps in the available memory, which can cause out-of-memory when you try to allocate an object larger than any of the blocks of available memory.

See here for more details.

Any object larger than 85,000 bytes will be placed on the large object heap, except for arrays of doubles for which the threshold is just 1000 doubles (or 8000 bytes).

Also note that 32-bit .Net programs are limited to a maximum of 2GB per object and somewhat less than 4GB overall (perhaps as low as 3GB depending on the OS).

Upvotes: 3

Related Questions