Windows program has big native heap, much larger than all allocations

Question

We are running a mixed mode process (managed + unmanaged) on Win 7 64 bit.

Our process is using up too much memory (especially VM). Based on our analysis, the majority of the memory is used by a big native heap. Our theory is that the LFH is saving too many free blocks in committed memory, for future allocations. They sum to about 1.2 GB while our actual allocated native memory is only at most 0.6 GB.

These numbers are from a test run of the process. In production it sometimes exceeded 10 GB of VM - with maybe 6 GB unaccounted for by known allocations.

We'd like to know if this theory of excessive committed-but-free-for-allocation segments is true, and how this waste can be reduced.

Here's the details of our analysis.

1. First we needed to figure out what's allocated and rule out memory leaks. We ran the excellent Heap Inspector by Jelle van der Beek and we ruled out a leak and established that the known allocations are at most 0.6 deci-GB.

2. We took a full memory dump and opened in WinDbg.

3. Ran !heap -stat It reports a big native heap with 1.83 deci-GB committed memory. Much more than the sum of our allocations!

_HEAP 000000001b480000
     Segments            00000078
         Reserved  bytes 0000000072980000
         Committed bytes 000000006d597000
     VirtAllocBlocks     0000001e
         VirtAlloc bytes 0000000eb7a60118

4. Then we ran !heap -stat -h 0000001b480000

heap @ 000000001b480000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    c0000 12 - d80000  (10.54)
    b0000 d - 8f0000  (6.98)
    e0000 a - 8c0000  (6.83)
...

If we add up all the 20 reported items, they add up to 85 deci-MB - much less than the 1.79 deci-GB we're looking for.

5. We ran !heap -h 1b480000

...
Flags:                00001002
    ForceFlags:           00000000
    Granularity:          16 bytes
    Segment Reserve:      72a70000
    Segment Commit:       00002000
    DeCommit Block Thres: 00000400
    DeCommit Total Thres: 00001000
    Total Free Size:      013b60f1
    Max. Allocation Size: 000007fffffdefff
    Lock Variable at:     000000001b480208
    Next TagIndex:        0000
    Maximum TagIndex:     0000
    Tag Entries:          00000000
    PsuedoTag Entries:    00000000
    Virtual Alloc List:   1b480118
    Unable to read nt!_HEAP_VIRTUAL_ALLOC_ENTRY structure at 000000002acf0000
    Uncommitted ranges:   1b4800f8
    FreeList[ 00 ] at 000000001b480158: 00000000be940080 . 0000000085828c60   (9451 blocks)
...

When adding up up all the segment sizes in the report, we get:

Total Size                               = 1.83 deci-GB
Segments Marked Busy Size                = 1.50 deci-GB
Segments Marked Busy and Internal Size   = 1.37 deci-GB

So all the committed bytes in this report do add up to the total commit size. We grouped on block size and the most heavy allocations come from blocks of size 0x3fff0. These don't correspond to allocations that we know of. There were also mystery blocks of other sizes.

6. We ran !heap -p -all. This reports the LFH internal segments but we don't understand it fully. Those 3fff0 sized blocks in the previous report appear in the LFH report with an asterisk mark and are sometimes Busy and sometimes Free. Then inside them we see many smaller free blocks.

We guess these free blocks are legitimate. They are committed VM that the LFH reserves for future allocations. But why is their total size so much greater than sum of memory allocations, and can this be reduced?

Answer 1

Well, I can sort of answer my own question.

We had been doing lots and lots of tiny allocations and deallocations in our program. There was no leak, but it seems this somehow created a fragmentation of some sort. After consolidating and eliminating most of these allocations our software is running much better and using less peak memory. It is still a mystery why the peak committed memory was so much higher than the peak actually-used memory.