Unmanaged memory leak in mscorwks help analyze

Question

Story:

We are facing unmanaged memory leak in our .NET 2.0 application. Process after start consume about 150MB (most of it is .NET managed, object states and so). After about 12 hours of running process consumed up to 800MB and after next 12 hours process have about 1.8GB of RAM. I just tried JetBrains .NET Memory Profiler, ANTS, .NET Memory Profiler (and maybe 2 next mem profiles available on market) none of this helps me because as i detect later our process consume that much memory in unmanaged area not managed. To detect this i used Perf monitor with counters: Private Bytes(Process) and # Bytes in All Heaps (.NET CLR Memory) where Private Bytes consume about 90% of all memory allocated by process. This is why i switch to unmanaged debugging.

DebugDiag: So i run debugdiag on process and get full dump, here is snapshot of it:

• mscorwks.dll (a known Windows memory manager) is responsible for 781,73 MBytes worth of outstanding allocations. These allocations appear to have originated from the following module(s) and function(s):

• ntdll.dll (a known Windows memory manager) is responsible for 98,24 MBytes worth of outstanding allocations. These allocations appear to have originated from the following module(s) and function(s):

Top 4 functions by allocation count

• mscorwks!EEHeapAlloc+15b -- 80 957 allocation(s)
• mscorwks!CLRMapViewOfFileEx+4a -- 4 171 allocation(s)

Top 4 functions by allocation size

• mscorwks!EEVirtualAlloc+15b -- 117,50 MBytes
• mscorwks!EEHeapAlloc+15b -- 15,03 MBytes

Interesting logs found:

Function details

Function mscorwks!EEVirtualAlloc+15b

• Allocation type Virtual memory allocation(s)
• Allocation Count 1471 allocation(s)
• Allocation Size 117,50 MBytes
• Leak Probability 73%

Function mscorwks!EEHeapAlloc+15b

• Allocation type Heap allocation(s)
• Allocation Count 80957 allocation(s)
• Allocation Size 15,03 MBytes
• Leak Probability 72%

Function mscorwks!CExecutionEngine::CheckThreadState+fe

• Allocation type Heap allocation(s)
• Heap handle 0x00000000`00000000
• Allocation Count 2 allocation(s)
• Allocation Size 304 Bytes
• Leak Probability 98%

Function mscorwks!CLRMapViewOfFileEx+4a

• Allocation type Virtual memory allocation(s)
• Allocation Count 4171 allocation(s)
• Allocation Size 0 Bytes
• Leak Probability 73%

I want someone to push me in right direction how can i found memory leak from this dump? I'm able to loaded dump into windbg and run standard set of windbg command but i dont know which one are the right commands to be able to isolate leak.

I can provide full dump if anyone want to help with this.

Answer 1

Looking at the dump file it does seem to be a managed leak, only not on the managed heap. The dump shows that the managed heap is quite small, but the loader heap is 1 GB. The process has over 35000 dynamic assemblies. I looked at a few of them and they appear to be serialization (XML and binary). Take a look at this blog post. It describes a similar issue.

Answer 2

My favorite way to debug .NET and Silverlight memory leaks is using the SOS extension. See here for a quick walkthrough: http://blogs.msdn.com/b/ricom/archive/2004/12/10/279612.aspx

What I generally do is:

1. !DumpHeap -stat to get a list of what objects exist in memory; this will usually indicate the source of the issue (e.g. if I see a million tiny objects that should only be used once and thrown away).
2. Once I know what type of object causes it, I dump a list of such objects using !DumpHeap and randomly get the roots of a few of them (!GCRoot). This will usually indicate what object is unexpectedly holding references to the leaky objects.

This assumes that you are dealing with a managed memory leak (referenced being kept where you do not expect it). It will not help much if you are dealing with an unmanaged memory leak but that is much less likely unless your application does a lot of manual unmanaged memory management (e.g. object marshalling for P/Invoke).

Answer 3

using weak references may reduce memory usage http://en.wikipedia.org/wiki/Weak_reference .. that's very general, but once it saved me almost few GB s