Why FREE block so large in heap

Question

I have an app built with C# and WPF, running on Windows 8. The C# code calls external C++ dll's using Interop. After a series of actions, memory usage up to near 2GB, so a memory leak is suspected. I created a dump of the process and try to analyze it with WinDbg, by command !address -summary, I got the following message:

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    482      7ff`4d11b000 (   7.997 Tb)           99.97%
<unknown>                              1208        0`7573d000 (   1.835 Gb)  65.64%    0.02%
Heap                                   1664        0`1e1ab000 ( 481.668 Mb)  16.82%    0.01%
Image                                  1294        0`19f98000 ( 415.594 Mb)  14.52%    0.00%
Stack                                   261        0`053ce000 (  83.805 Mb)   2.93%    0.00%
Other                                    23        0`001d8000 (   1.844 Mb)   0.06%    0.00%
TEB                                      87        0`000ae000 ( 696.000 kb)   0.02%    0.00%
PEB                                       1        0`00001000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            2634        0`922ae000 (   2.284 Gb)  81.69%    0.03%
MEM_IMAGE                              1787        0`1b38d000 ( 435.551 Mb)  15.21%    0.01%
MEM_MAPPED                               90        0`05807000 (  88.027 Mb)   3.07%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                509      7ff`4d1ae000 (   7.997 Tb)           99.97%
MEM_COMMIT                             3346        0`9205e000 (   2.282 Gb)  81.61%    0.03%
MEM_RESERVE                            1165        0`20de4000 ( 525.891 Mb)  18.37%    0.01%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         1577        0`73295000 (   1.799 Gb)  64.36%    0.02%
PAGE_EXECUTE_READ                       183        0`1212e000 ( 289.180 Mb)  10.10%    0.00%
PAGE_READONLY                           941        0`082f9000 ( 130.973 Mb)   4.57%    0.00%
PAGE_READWRITE|PAGE_WRITECOMBINE         24        0`03aad000 (  58.676 Mb)   2.05%    0.00%
PAGE_EXECUTE_READWRITE                  131        0`00bcc000 (  11.797 Mb)   0.41%    0.00%
PAGE_READWRITE|PAGE_GUARD                87        0`00191000 (   1.566 Mb)   0.05%    0.00%
PAGE_NOACCESS                           399        0`0018f000 (   1.559 Mb)   0.05%    0.00%
<unknown>                                 1        0`00004000 (  16.000 kb)   0.00%    0.00%
PAGE_EXECUTE                              2        0`00003000 (  12.000 kb)   0.00%    0.00%
PAGE_WRITECOPY                            1        0`00002000 (   8.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                     a1`9f910000      754`5f4fc000 (   7.330 Tb)
<unknown>                                a1`4fc30000        0`17ffc000 ( 383.984 Mb)
Heap                                     a1`1c421000        0`00dff000 (  13.996 Mb)
Image                                     0`6c991000        0`0174e000 (  23.305 Mb)
Stack                                    a1`160c0000        0`000fb000 (1004.000 kb)
Other                                    a1`4e480000        0`00181000 (   1.504 Mb)
TEB                                     7f5`fee0c000        0`00002000 (   8.000 kb)
PEB                                     7f5`ff14c000        0`00001000 (   4.000 kb)

Why the Free block is so large? Within the summary, any clue where memory leak could be?

Answer 1

What you describe is a common issue, not only for testers, also for developers. You never know how much memory is really free until you understand the whole picture.

I use the analogy of a barkeeper to explain the issue. That barkeeper was engaged for a very big party. He does not own enough glasses, so he contacts a glass rental company. That glass rental company has 8000 glasses to rent.

The barkeeper orders 2000 glasses. At the beginning of the party, all glasses are empty. But who can give this statement? From the glass rental company point of view, the 2000 glasses are gone - they cannot be ordered any more. So the glass rental company considers them as "in use", whereas the barkeeper considers them as "not in use".

The party begins, waiters receive orders and the barkeeper mixes drinks. At some point in time, the barkeeper has given 500 glasses to the waiters. From his point of view, 1500 glasses are empty and 500 are full.

However, these 500 glasses might not be full. Some guests may have been very thirsty and already emptied their glasses. So in reality, 200 of those 500 glasses may be empty and only 300 glasses are full.

So, it depends on whom you ask:

• glass rental company: 2000 glasses are rented
• barkeeper: 500 glasses have been given away and not returned
• guests: 300 glasses are full

The same applies to virtual memory of a program, just the actors are different:

• glass rental company = the operating system
• barkeeper = heap managers, such as .NET heap manager or Windows Heap manager (C++ and others)
• guests = application logic

With !address you ask the operating system. If it says "I have given 1.8 GB to .NET", then that's a correct statement that contributes towards the understanding of reality.

However, you don't know what .NET says. You need to ask it. !dumpheap -stat would be an appropriate question. That answer will include some statements about objects of type Free.

And even if you see a byte[1000000], which is considered as "in use" by .NET, then those 1 MB may be partially empty from application logic point of view. If that byte[] is a buffer, there's typically some int length property which tells the application how much of that buffer is actually in use. The rest could be considered as "free".

Conclusion

You need to ask the right questions on the right level to figure out whether memory is in use or not.

Why the Free block is so large?

Because that's the answer of the glass rental company. There are plenty of glasses to rent (6000 are left).

Within the summary, any clue where memory leak could be?

A leak is indicated by growth. You cannot identify it from a single snapshot. You need to monitor it over time. Which memory is growing, which is not? Unfortunately, it's not possible to say anything from the given information.

Answer 2

In theory, a 64-bit process can access 16 exbibytes of virtual memory, but modern CPUs and OSes generally limit this. Windows allows 8 terabytes to be addressed. Please note that all 64-bit processes have their own private 8 terabyte virtual memory block, regardless of how much physical memory you have in your computer. The <unknown> block indicates memory used via virtual allocations (among others), which includes .NET memory. This is the 1.835 gigabytes of virtual memory that you mentioned. A terabyte is 1,024 gigabytes, which is only a very small fraction of the total addressable virtual memory of 8 terabytes. As a result, the Free block remains very large. This is quite typical in 64-bit programs since not many programs use multiple terabytes of memory. Try this experiment with a 32-bit program to see much different results.