.NET Out Of Memory Exception - Used 1.3GB but have 16GB installed

Question

I am getting an Out Of Memory exception in my C# application when the memory usage for the application goes over about 1.3GB.

I had this same problem on a 32-bit machine with 3GB of memory and it made sense back then. But now I upgraded the hardware to a 64-bit machine with 16GB memory using a high-end motherboard and high-end RAM, but the Out Of Memory exception still occurs after 1.3GB!

I know that there are no single objects over 2GB and 1.3 is less than 2GB anyway, so the built-in MS 2GB limit on a single object is not likely to be the problem.

It seems like there is a Windows kill-switch of some sort when an app reaches a certain memory usage threshold. Then there should be a way to configure this. Is it in the registry perhaps?

Any help will be greatly appreciated!

Answer 1

This problem has always annoyed me and, in my opinion, Microsoft really needs to do something about it. Twenty years ago it wasn't a problem because large amounts of memory just weren't accessible, that problem disappeared prior to 2009 however.

Even small to medium sized companies in 2009 had servers with 64 GB of RAM, I know I worked for one. At that same time 16 GB of RAM was becoming common on desktops. Today everything has changed and I can build a brand new desktop with 128 GB of RAM for less than $1,000. Even laptops with 64 GB of RAM are available and just now I customized a Dell laptop with 32 GB or RAM for $1250. Suffice it to say that the days of constrained RAM are long behind us and "modern" development tools have fallen way behind.

No this isn't just a rant on my part though it is at least a partial rant. I am in the unenviable position of having to develop some code for a 32 bit architecture, in .Net 4.8 no less, and part of my requirements involve using lots of RAM. My testing had revealed that in 32 bit mode on my desktop, C# was only able to allocate 1.3 GB of RAM just as the OP found. I was a little surprised to see this problem still persists a decade later.

Thanks to the good folks here (including but not limited to Julian Miller - why was his answer downvoted anyway) my 32 bit application, within the confines of the development environment even, can now allocate almost 4 GB of RAM (or 3.87 GB to be exact).

I am using Visual Studio 2022 and the key to solving the problem was the combination of Desty's and Julian Miller’s answers. It was as simple as putting the following in the post build event in my project’s property pages.

if exist "$(DevEnvDir)..\tools\VsDevCmd.bat" (
   call "$(DevEnvDir)..\tools\VsDevCmd.bat"
   editbin /largeaddressaware "$(TargetPath)"
)

In order to try to get an accurate assessment of the amount of RAM a 32 bit C# application would be able to allocate, I created a Windows console application test project with minimal code. I ran it on my desktop which has 16 GB of RAM with 10+ GB free at the time I ran it.

My test code follows, if you’d like to verify the results independently. Note that in order to allocate the maximum amount of RAM I had to run the code after restarting Visual Studio on occasion, maybe this is due to memory fragmentation or a leak. Note also that I realize the code looks strange but once I allocated 3 large segments of RAM I was unable to allocate another large segment so for each subsequent allocation I had to allocate numerous progressively smaller segments. I'm no memory allocation expert so I can only assume that has to do with memory fragmentation.

static void Main(string[] args)
{
    long totalAllocated;
    List<IntPtr> pointers = new List<IntPtr>(3000000);

    pointers.Add(Marshal.AllocHGlobal(1073741823));
    pointers.Add(Marshal.AllocHGlobal(1073741823));
    pointers.Add(Marshal.AllocHGlobal(1060766923));

    totalAllocated = 3208250569;

    for (int i = 0; i < 8; i++)
        pointers.Add(Marshal.AllocHGlobal(102400000));

    totalAllocated += 819200000;

    for (int i = 0; i < 7; i++)
        pointers.Add(Marshal.AllocHGlobal(10240000));

    totalAllocated += 71680000;

    for (int i = 0; i < 44; i++)
        pointers.Add(Marshal.AllocHGlobal(1024000));

    totalAllocated += 45056000;

    for (int i = 0; i < 135; i++)
        pointers.Add(Marshal.AllocHGlobal(102400));

    totalAllocated += 13824000;

    for (int i = 0; i < 200; i++)
        pointers.Add(Marshal.AllocHGlobal(10240));

    totalAllocated += 2252800;

    for (int i = 0; i < 300; i++)
        pointers.Add(Marshal.AllocHGlobal(1024));

    totalAllocated += 307200;

    for (int i = 0; i < 300; i++)
        pointers.Add(Marshal.AllocHGlobal(100));
    
    totalAllocated += 30000;

    for (int i = 0; i < pointers.Count; i++)
    {
        Marshal.FreeHGlobal(pointers[i]);
        pointers[i] = IntPtr.Zero;
    }
}

For the naysayers who will point to 64 bit mode, my test code could allocate a mere 5 GB of RAM in 64 bit mode. For the other naysayers, who need to stop it, who will claim no one needs to use that much RAM... we are living in the era of big data and our customers want to mine that data on their 128 GB or 256 GB desktops. The era of 4 GB has long since passed. Sadly I suspect Microsoft has no interest in making such a change to dot net.

Also, in the interest of being as thorough as possible following is a screenshot of the VS 2022 property pages of my test project showing the post build code which I had to include to enable allocation of the additional RAM.

Answer 2

If you just want to provide access to more memory for a specific 32 bit app outside of VS, there is a useful application at https://github.com/cybertechniques/site/blob/master/analysis_tools/cff-explorer/index.md which will set the apps headers for you.

• Download "CFF Explorer" from the NT Core site or CFF Explorer file.
• Launch "CFF Explorer" and open the EXE (e.g. C:\Users\dennisg\AppData\Local\Apps\2.0\MHV1WDDC.958Y0ELE416.L7Y\myapp.exe)
• List item

In the left side of the window, choose the "File Header" section under "Nt Headers" Look on the right side for the row starting with "Characteristics", and the "Click here" field. Click there. (A context menu appears for the next part). Select/enable the checkbox for "App can handle >2gb addresses" and click 'OK'. Save the file (File > Save).

Answer 3

if you're searching for the tools folder in a newer Version of Visual Studio (2017, 2019), I've got it under

C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\Tools

and the files' name is VsDevCmd.bat from Visual Studio 2017 on and later.

Answer 4

If you have 32-bit Windows, this method is not working without following settings.

1. Run prompt cmd.exe (important : Run As Administrator)
2. type bcdedit.exe and run
3. Look at the "increaseuserva" params and there is no then write following statement
4. bcdedit /set increaseuserva 3072
5. and again step 2 and check params

We added this settings and this block started.

if exist "$(DevEnvDir)..\tools\vsvars32.bat" (
   call "$(DevEnvDir)..\tools\vsvars32.bat"
   editbin /largeaddressaware "$(TargetPath)"
)

More info - command increaseuserva: https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/bcdedit--set

Answer 5

There is no difference until you compile to same target architecture. I suppose you are compiling for 32 bit architecture in both cases.

It's worth mentioning that OutOfMemoryException can also be raised if you get 2GB of memory allocated by a single collection in CLR (say List<T>) on both architectures 32 and 64 bit.

To be able to benefit from memory goodness on 64 bit architecture, you have to compile your code targeting 64 bit architecture. After that, naturally, your binary will run only on 64 bit, but will benefit from possibility having more space available in RAM.

Answer 6

Its worth mentioning that the default for an 'Any CPU' compile now checks the 'Prefer 32bit' check box. Being set to AnyCPU, on a 64bit OS with 16gb of RAM can still hit an out of memory exception at 2gb if this is checked.

Answer 7

As already mentioned, compiling the app in x64 gives you far more available memory.

But in the case one must build an app in x86, there is a way to raise the memory limit from 1,2GB to 4GB (which is the actual limit for 32 bit processes):

In the VC/bin folder of the Visual Studio installation directory, there must be an editbin.exe file. So in my default installation I find it under

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\editbin.exe

In order to make the program work, maybe you must execute vcvars32.bat in the same directory first. Then a

editbin /LARGEADDRESSAWARE <your compiled exe file>

is enough to let your program use 4GB RAM. <your compiled exe file> is the exe, which VS generated while compiling your project.

If you want to automate this behavior every time you compile your project, use the following Post-Build event for the executed project:

if exist "$(DevEnvDir)..\tools\vsvars32.bat" (
   call "$(DevEnvDir)..\tools\vsvars32.bat"
   editbin /largeaddressaware "$(TargetPath)"
)

Sidenote: The same can be done with the devenv.exe to let Visual Studio also use 4GB RAM instead of 1.2GB (but first backup the old devenv.exe).

Answer 8

Is your application running as a 64 or 32bit process? You can check this in the task manager.

It could be, it is running as 32bit, even though the entire system is running on 64bit.

If 32bit, a third party library could be causing this. But first make sure your application is compiling for "Any CPU", as stated in the comments.

Answer 9

It looks like you have a 64bit arch, fine -- but a 32bit version of the .NET runtime and/or a 32bit version of Windows.

And as such, the address space available to your process is still the same, it has not changed from your previous setup.

Upgrade to both a 64bit OS and a 64bit .NET version ;)