CODEWITHSUNDEEP
c#.netmemory-managementout-of-memory
DarkWanderer
DarkWanderer

Reputation: 8866

System.OutOfMemoryException is thrown while still having free VM space

Before asking, I'll make a small disclaimer: Yes, I am aware differences between Virtual Memory, Physical Memory and Working Set. All the numbers below refer to the virtual memory.

The situation is as follows: We have a 32-bit C# app, which imports x86 C++ libraries (with plenty of native dependencies, so migrating to x64 is not an option at the moment). The app loads a large dataset through the unmanaged component, and then tries to display it (report).

However, when the dataset is particularly large, an OutOfMemory exception is thrown upon adding an item to the list, like in code below:

No surprise here. However, what is surprising is the fact that the application still has around 280MB free VM.

When debugging pure unmanaged apps (C++), it was never the case - the bad_alloc could be only achieved if there was no free VM left, or if there was no free address space chunk of sufficient size.

Hence, the question - what can be the reason of this? I understand how to fight the issue - the unmanaged components really do eat a lot of memory, as well as create a lot of fragmentation - but what is the reason for OutOfMemoryException appearing so early?

Code in question looks like this:

List<Cell> r = new List<Cell>(cols);
for (int j = 0; j < cols; j++)
{
    r.Add(new CustomCell()); // The exception is thrown on this line
}

At the moment of exception, the list (in one occurence) had 85 items and it's capacity was 200-something (the number of columns, as indicated in constructor). So, the exception most likely happened in CustomCell allocation. CustomCell object has many fields, but is certainly less than 1KB in total. The 280MB of free memory are located in chunks from 64KB to 14MB - so there should be plenty of space to allocate it.

Upvotes: 0

Views: 682

Answers (3)

Hans Passant
Hans Passant

Reputation: 942000

there were chunks up to 14Mb free. The operation which was failing was creating an object less than 1KB total

14 MB is definitely close to the danger-zone. The way the GC allocates VM has nothing to do with the object size. The GC heap is created from chunks of VM called "segments". The segment size is 2 MB when a program starts out but the GC dynamically grows the allocation of new segments when the program uses a lot of memory. Clearly you use a lot of memory. There isn't anything you can do to affect the VM allocation or avoid VM address space fragmentation.

Clearly you are way too close to the VM limit of a 32-bit process. You'll need to either drastically revise your code so you can make do with less. Or you need to put a 64-bit operating system on your list of prerequisites. Which can provide 4 gigabytes of VM address space to a 32-bit process. You'll need an extra build step to take advantage of it, described in this answer.

Upvotes: 2

Bassam Alugili
Bassam Alugili

Reputation: 17003

You have to take care also when you have 64 bit application the maximum array size is 2 GB check if you are loading your report table in an array.

More info: OutOfMemoryException on declaration of Large Array

This also might be intersseting for you: http://social.msdn.microsoft.com/Forums/en-US/1a12abaa-50bd-4d28-b3c1-9de06a1488e9/how-to-create-an-extremely-large-arrayobject-2-gb-without-using-jagged-arrays-

Upvotes: 0

spender
spender

Reputation: 120498

EDIT

Written before you added your code. Looks like you're already doing this:

List instances have a capacity. If you exceed the capacity, a growing algorithm is invoked which doubles the capacity of the list.

//decompiled from List<T>
private void EnsureCapacity(int min)
{
  if (this._items.Length >= min)
    return;
  int num = this._items.Length == 0 ? 4 : this._items.Length * 2;
  if ((uint) num > 2146435071U)
    num = 2146435071;
  if (num < min)
    num = min;
  this.Capacity = num; //causes a copying of src array to a new array
}

This can lead to unexpected memory allocation. If you are able to anticipate the final size of the list, allocate it up front and avoid the doubling:

var myList = new List<SomeType>(expectedCapacity)

Upvotes: 0

Related Questions