Memory leak issue with pinned object heap

Question

Issue and attempted debug I have a fairly large enterprise application I am managing, it has a memory leak which after investigating actually looks to be a whole bunch of empty memory because pinned objects are preventing the garbage collector from working.

I took a memory dump when the application was running at over 12GB total memory on Azure. The actual memory recorded inside the app is far lower: 200MB of actual memory used

I can see that the application has a 9.5GB generation 2 garbage collector heap but its 99% empty memory, and its just tons of pinned objects (about 3000) that are scattered all over the heap preventing any compaction. Pinned objects

The memory leaking type is pinned is a byte[8192], it always has every single byte inside set to 0. Therefore I can't try to decode the byte[] into something I can understand and it appears that it is not possible to get a retention tree on pinned objects to see where they are referenced (or at least this one does not seem to have one).

From my understanding of how c# memory leaks work and particularly pinned objects, I believe a pinned object can only be created if someone uses the fixed keyword inside an unsafe block. The class with this unsafe would typically be marked as disposable so that the fixed object can be cleaned up before it is garbage collected. And you would expect to dispose/ add using var..... up the hierarchy.

I have double checked that project does not allow unsafe code, I have also added the following to force me to dispose everything with a disposable interface (CA2000) this did highlight. It seems that everything is disposed of, I do not have any fixed in my code and yet I am getting pinned objects everywhere that do not get removed.

Firstly it would be good to have confirmation I have not misunderstood anything above. Secondly, how can I actually track down what is creating these arrays.

Project Details .net core 8.0 Because the project is used by many customers and its code for a company we supply I do not believe its reasonable or safe to share either the memory dumps or source code, and the issue seems to only be problematic at load.

Below I have added all the nuget packages we use, as I do not allow any unsafe code, I expect the actual leak is happening in one of these but without a stack its hard to tell where its being created.

We do have the same code deployed to 2 instances and I am confident that the memory leak does not happen inside the service bus handlers or cosmos database as a server we use has no API calls made to it, but does still use service bus and cosmos a lot without any apparent leak. I am currently thinking the leak is therefore a part of the API stack to give you a summary of how things typically work:

Server 1 (has leak) - API -> Read from Cosmos -> Do reflection -> Write to Cosmos -> Publish service bus message

Server 2 (no leak) - Handle service bus message -> Read from Cosmos -> Write to Cosmos

Same code is deployed to both, but we do not call the API on server 2.

I mention in there that we do some reflection, I am just pointing that out because I noticed we have a bit of reflection.Emit which looks far more problematic than our regular code. Would be good if anyone can confirm if the below code could in anyway cause this (there are no disposables here so I think not, also the output of the below code is stored and not regenerated so only runs 8 times in total, I don't see it creating 3000 objects):

var typeSignature = this.GetType().Name + "Suffix";`
var assemblyBuilder = AssemblyBuilder.DefineDynamicAssembly(new AssemblyName(Guid.NewGuid().ToString()), AssemblyBuilderAccess.Run);
var moduleBuilder = assemblyBuilder.DefineDynamicModule("MainModule");
var typeBuilder = moduleBuilder.DefineType(typeSignature,
TypeAttributes.Public |
TypeAttributes.Class |
TypeAttributes.AutoClass |
TypeAttributes.AnsiClass |
TypeAttributes.BeforeFieldInit |
TypeAttributes.AutoLayout,
null);

This has been a hard issue to solve, I have tried many things before posting but I am now out of ideas so any advice would be appreciated?

Edit with more Info I have found that I have 29937 instances of this Byte[][Pinned Handle] I have looked for a similar count of objects hoping that I could find a class that was being created 1-1 ratio of creating these.

I have added a screenshot that shows that I have about 29k-30k of each of these types, but I do not know why they are here or if they can even cause this bug.

IOCompletionCallbackHelper
FileSystemWatcher+ASyncReadState
ThreadPoolBoundHandle
PreAllocatedOverlapped
ThreadPoolBoundHandlerOverlapped

Memory leak issue with pinned object heap

Answers (1)

Related Questions