ASPX.NET application out of memory exception for no reason

Question

Here is the deal: when my web server starts up, it creates a couple of lengthy (20M of elements) arrays with really small objects (like 1-2-3 ints). The accumulative size of any individual array is NOT larger than 2GB (the limitation of CLR, see the link below for some details). The w3wp.exe does grow in memory usage close to 2GB (never more than that). The code is compiled in Any CPU platform mode and run on Windows 7 x64 with 8GB of RAM.

What on earth makes it to throw OutOfMemoryException while creating my lists? Does it make any difference if I host the process thru IIS or VS? This appears not happening is PROD but I am experiencing this on my dev machine all the time. (Will try to restart now...)

This may be related but I don't seem to have objects that big: Very large collection in .Net causes out-of-memory exception

EDIT: It does make difference to run in IIS or VS - don't see that happening when the process is started in IIS. So could it be VS debugger limitation?

Answer 1

Based on your updated question, it's obvious that Visual Studio does not run in 64 bit mode. So your limitation is 2GB under Visual Studio.

This post probably contains some code helpful to prove this fact: How to detect Windows 64-bit platform with .NET?

Answer 2

Check if you have apppool recycling threshold set to 2GB http://technet.microsoft.com/en-us/library/cc732519%28WS.10%29.aspx

Answer 3

Probably memory allocations are not optimized (i.e. done in small steps and resizes). This has a potential to fragment the heap such, that there is no longer enough contigious free space to store the 'semi-large' array.

That allocation fails, and this situation is by definition OOM, even though plenty fragments of heap might be available. Usually, excessive use of linq can cause this; at a certain point deferred execution looses it's appeal, and you can buy a lot of performance/resources by doing one or two '.ToList()' at strategic places (in my experience, often close to the beginning of your generating process, where the bulk of the data arrives).