.net unmanaged memory leak

Question

I have a windows service to receive emails that uses OpenPop. However memory usage goes up to 8G about 3 days after a restart. Operation staff gives me a dump file, So I use windbg to analysis it.

When I run !address -summary I got:

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    468      7fd`942f6000 (   7.991 TB)           99.88%
Heap                                    645        1`c3f95000 (   7.062 GB)  72.92%    0.09%
<unknown>                              1347        0`9305f000 (   2.297 GB)  23.72%    0.03%
Image                                  1985        0`0d28d000 ( 210.551 MB)   2.12%    0.00%
Stack                                   366        0`077c0000 ( 119.750 MB)   1.21%    0.00%
Other                                    11        0`001c4000 (   1.766 MB)   0.02%    0.00%
TEB                                     122        0`000f4000 ( 976.000 kB)   0.01%    0.00%
PEB                                       1        0`00001000 (   4.000 kB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            1725        2`5b3b9000 (   9.426 GB)  97.33%    0.12%
MEM_IMAGE                              2692        0`0f249000 ( 242.285 MB)   2.44%    0.00%
MEM_MAPPED                               60        0`016f8000 (  22.969 MB)   0.23%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                468      7fd`942f6000 (   7.991 TB)           99.88%
MEM_COMMIT                             3373        2`10e49000 (   8.264 GB)  85.33%    0.10%
MEM_RESERVE                            1104        0`5aeb1000 (   1.421 GB)  14.67%    0.02%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         1528        2`0092c000 (   8.009 GB)  82.70%    0.10%
PAGE_EXECUTE_READ                       308        0`0b666000 ( 182.398 MB)   1.84%    0.00%
PAGE_READONLY                           924        0`0323a000 (  50.227 MB)   0.51%    0.00%
PAGE_WRITECOPY                          321        0`012f3000 (  18.949 MB)   0.19%    0.00%
PAGE_EXECUTE_READWRITE                  112        0`005d2000 (   5.820 MB)   0.06%    0.00%
PAGE_READWRITE|PAGE_GUARD               122        0`0023c000 (   2.234 MB)   0.02%    0.00%
PAGE_EXECUTE_WRITECOPY                   57        0`00178000 (   1.469 MB)   0.01%    0.00%
PAGE_EXECUTE                              1        0`00004000 (  16.000 kB)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                      5`2d890000      7f9`6d640000 (   7.974 TB)
Heap                                      0`25d40000        0`00fd0000 (  15.813 MB)
<unknown>                                 0`32e89000        0`0eb07000 ( 235.027 MB)
Image                                   7fe`f466a000        0`01338000 (  19.219 MB)
Stack                                     0`19c60000        0`000fc000 (1008.000 kB)
Other                                     0`00de0000        0`00181000 (   1.504 MB)
TEB                                     7ff`ffdb8000        0`00002000 (   8.000 kB)
PEB                                     7ff`fffdf000        0`00001000 (   4.000 kB)

and !eeheap -gc I got:

Number of GC Heaps: 1
generation 0 starts at 0x000000051da29040
generation 1 starts at 0x000000051d891000
generation 2 starts at 0x0000000001281000
ephemeral segment allocation context: none
 segment     begin allocated  size
0000000001280000  0000000001281000  000000000b1eae80  0x9f69e80(167157376)
0000000031990000  0000000031991000  0000000032e68ee0  0x14d7ee0(21855968)
000000007fff0000  000000007fff1000  0000000081a99330  0x1aa8330(27951920)
000000008fff0000  000000008fff1000  0000000099d1bb00  0x9d2ab00(164801280)
000000009fff0000  000000009fff1000  00000000a77989c0  0x77a79c0(125467072)
000000051d890000  000000051d891000  000000052352fe40  0x5c9ee40(97119808)
Large object heap starts at 0x0000000011281000
 segment     begin allocated  size
0000000011280000  0000000011281000  0000000019236c28  0x7fb5c28(133913640)
0000000048fa0000  0000000048fa1000  0000000050a87908  0x7ae6908(128870664)
0000000050fa0000  0000000050fa1000  000000005387c418  0x28db418(42841112)
00000000afff0000  00000000afff1000  00000000b3bf0840  0x3bff840(62912576)
Total Size:              Size: 0x39fd2518 (972891416) bytes.
------------------------------
GC Heap Size:            Size: 0x39fd2518 (972891416) bytes.

From these two commands, there is some unmanaged memory leak. How can I find out what is in the unmanaged memory and by what method?

Answer 1

There is no evidence yet, that your situation is a native memory leak. Yes, you have 8 GB in native heaps. But you also have 900 MB in managed heaps which could hold the native objects alive.

I suggest using sos and starting with !dumpheap -stat if you really want to go with WinDbg. Otherwise use a .NET memory profiler. Many people have a JetBrains dotMemory license because they use R# Ultimate. It's much easier to use than WinDbg and has better support for comparing snapshots over time.

Answer 2

As far as I can see there is no breakdown into memory usage of individual .net components. To do this you would need to load the SOS debugger extension. The article Hunting .NET memory leaks with Windbg by André Snede Kock provides some more details.

If you are new to memory debugging I would probably also recommend a commercial tool, like dotMemory or ANTS, since they tend to be easier to use.

This should give you a list of .Net objects, how many instances are alive, and how much memory they use. I usually try to look for unexpected things, typically if there are more objects alive than I would expect, and then try to find out why. This helps with detecting problems in managed code, and I would say this is the most likely problem.

Another possibility is that there is a un-managed memory leak in some libary you use, but then you would need to contact the vendor of that library to get it fixed.