Analysing a crash dump on Windows, most of the memory is "busy internal", what is it?

Question

I'm running a 32bit application and it fails on shortage of memory (bad_alloc is thrown). However, when I'm looking in the dump file I can see that most of the memory is in "busy internal". Moreover, it has 3GB of committed memory. To put it as questions

1. Does it make sense that an app has so much "busy internal" memory?
2. Is it something I can control and/or configure?
3. Can I trigger some sort of memory consolidation of defragmentation?

DebugDiag:

Virtual Memory Analysis 
Virtual Memory Summary 
Size of largest free VM block   6.91 MBytes 
Free memory fragmentation       92.S6% 
Free Memory                     96.78 MBytes (2.36% ofT0ta1 Memory) 
Reserved Memory                 856.05 MBytes (20.9% ofT0ta1 Memory)
Committed Memory                3.07 GBytes (76.7400 ofT0ta1 Memory)
Total Memory                    4 GBytes 
Largest free block at           0x00000000'ff1f6000 

Partial output of WinDBG (!heap -a <heap>)

e3268eee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e32aaee8:  42008 . 21ff8 [101] - busy (21ff0) Internal 
e32cceee:  21ff8 . 22008 [101] - busy (21ff8) Internal 
e32eeee8:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e331eeee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e3352008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e3394eee:  41ff8 . 22008 [101] - busy (21ff8) Internal 
e33b6008:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e33d8eee:  21ff8 . 22008 [101] - busy (21ff8) Internal 
e33faee8:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e341ceee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e345eee8:  42008 . 21ff8 [101] - busy (21ff0) Internal 
e348eeee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e34c2008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e3504eee:  41ff8 . 42008 [101] - busy (41ff8) Internal 
e3546008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e3588eee:  41ff8 . 22008 [101] - busy (21ff8) Internal 
e35aaee8:  22008 . 21ff8 [101] - busy (21ff0) Internal 
e35cceee:  21ff8 . 42008 [101] - busy (41ff8) Internal 
e36eeee8:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e365eeee:  41ff8 . 42008 [101] - busy (41ff8) Internal 
e3692008:  42008 . 41ff8 [101] - busy (41ff0) Internal 
e36d4eee:  41ff8 . 22008 [101] - busy (21ff8) Internal 

UPDATE

Adding some more info in this - finding the total size of busy-internal

************************************************************************************************************************
                                              NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key                   : 0xc6ab7f8e
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
01590000 00000002 2995168 2981356 2994956  27250  2871   211    4   1db2   LFH
01bb0000 00001002    1292    536   1080     46     5     2    0      0   LFH
-----------------------------------------------------------------------------

I ran !heap -a 01590000 and saved the output to a log. Then I summed up the results, categorized by 'free', 'busy', 'busy-internal' and these were my results

busy 1.106 MB
busy-internal 2883.606 MB
free 26.612 MB
uncategorized 26.612 MB
total bytes 3080648904

This result agrees with the size of the memory shows, for both the total and free memory. It is clear that most of the memory is marked as "internal" for some vague reason.

Answer 1

Busy vs. free

A heap by the Windows Heap Manager is divided in segments and those segments are divided in blocks. Each block can basically have two states: free and busy, where busy means "in use". Assuming C++, int* pi = new int; gives you a busy block and delete pi; turns it back into a free block.

Does it make sense that an app has so much "busy internal" memory

"An app" - we don't know. Your app? Maybe. It depends on how often a new statement is invoked.

Is it something I can control and/or configure?

Sure. But you don't want to and you shouldn't. You can:

1. try to get rid of all new statements in your application. Question is: where do you want to store your data instead? Typically, you can exchange memory usage against CPU cycles and/or disk operations. E.g. you could write all data into a file and only read that portion from the file which is currently needed.
2. call VirtualAlloc() and then managing the memory yourself. This will reduce busy blocks, because you bypass the Windows Heap Manager completely. However, you would need to implement your own Heap Manager, which may result in something identical to busy blocks.
3. Reduce the overall available memory using GFlags. That way your app will fail earlier but you will have less busy blocks. Not helpful, I just took your request literally.

Can I trigger some sort of memory consolidation of defragmentation?

No. The Windows Heap Manager is sorting blocks by size already and reusing regions of the same size for the next new statement of the same size.

What you should consider:

• check your app for memory leaks. If this much memory is unexpected, then a memory leak can explain it. Try UMDH or a commercial tool, Valgrind if your app also runs on Linux.
• compile your app as 64 bit. You will have so much more memory available.
• if you compile your app in debug mode, switch to release mode. Release builds will use a different Heap Manager than debug builds. Also, optimization may reduce memory usage.
• if you make heavy use of the PImpl idiom, switching from arrays of pointers to arrays of data may help (also with performance).

Internal

As @Neitsa has once reverse-engineered, the Internal output corresponds to the HEAP_ENTRY_VIRTUAL_ALLOC flag.

IMHO, this means that the heap manager has decided to not manage this allocation and has forwarded the allocation to VirtualAlloc() instead. One consequence of this is that you won't have user mode stack traces for those allocations.