Azure Web App Memory Leaks Research

Question

QUESTION: On one of my web projects I've come across some difficult problem on web app. We have used Azure Web App services for our system. And once upon a time we have catch the most headache for any big app - memory leak. When You work with simple desktop app or publish Your app on virtual machine - there are a lot of different ways - how to catch leak. You can install different monitor and diagnose solutions and catch it in real time.

But What You should do, when you work with Azure Web App? (Or any other web service, that really chops off your diagnose opportunities). The best way - get dump and research it. You whould see this recommendation on any site, comment and blog. But You have to spent many many hours to find any information HOW you can do that.

So I just wanna make extract from all tons of information about web app memory leak research. Offcourse I'll give the most information in relation to Azure, but local dump research information You can apply to any other platform.

Answer 1

ANSWER:

HOW recognize memory leak

Lags on app - not best and safe answer) You should monitor Your diagnostic statistic on azure portal (more info here - https://learn.microsoft.com/en-gb/azure/monitoring-and-diagnostics/)

HOW catch dump from Azure Web App.

Every Azure Web App has it's own sandbox - an environment. You can access It with ".scm" suffix. For example if you site domain is "latvianmarksman.azurewebsites.net", then your sanbox address: "latvianmarksman.scm.azurewebsites.net". You should pass Your azure credentials to login. On Azure Web App Sandbox You can find different usefull info:

1. Tools > Support. You can find here events, diagnostic graphs, and MAKE CURRENT MEMORY DUMP (Analyze > Metrics). Azure will spent some time for getting dump, and make simple Analyze. You can download dump and analyse report on "Analyze > Diagnostics" page;

2. Debug > Cmd. You can find any local file of Your Web App. Usefull when You write your custom logs locally or when You have problems with web app size;

3. Proccess Exporer. This tab can give You information about main proccesses of Web App Sandbox, also on this tab You can get and download dump too;

4. Site Extentions. You can install different Site Extentions for reserach Your problem;

5. Tools > Diagnostic Dump. Third way to make dump;

WHAT TO DO with Azure Web App dump.

When Your problem is simple - You can find answer in Web App Events, or in simple Dump Analize; But to solve more complex problems You should make real deep dump analyze. The most forward and right way: use Win Dbg.

Firstly lets talk about alternative sources of information:

1. You can take chopped but sometimes usefull info from Azure Analyze
   Report (download in "tools>support" tab, near dump lynk).
2. DebugDiag2 - good tool for downloaded dump analyzing;
3. Also You can use MS Visual Studio to get simple analyze of dump.
   • You can analyze only managed memory;
   • You can analyze only on professional or enterprise version of MS Studio (not free Community);
4. Azure Web App diagnostics: only azure web app counters info.
   • Azure refreses this counters not so frequently;
   • Azure gives approximate information;
5. Azure Sql Diagnostics: only azure sql counters info.
   • Azure refreses this counters not so frequently;
   • Azure gives approximate information;
   • If You need counters values right now, use next script:
     SELECT sqltext.TEXT, req.session_id, req.status, req.command, req.cpu_time, req.total_elapsed_time, req.* FROM sys.dm_exec_requests req CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS sqltext

But anyway the most full information about memory leaks of any difficulty You can catch using WinDbg; This tool combines high level interface and low level analyzing power) WinDbg uses a lot of libraries but the most important of them: -netext

What should we do with dump:

1. Understand which memory (managed or unmanaged leaks)

   • Managed memory leaks, usually means garbage collector problems, when objects doesn't destructs correctly. On this leaks You will find a lot of objects in high level cache;
   • Unmanaged memory leaks, usually means uncorrect release of unmanaged resources. On this leaks You will find irregular use of heap; Just dowload Azure Dump Analyze and find GC section and Heap sections, analyze them and specify leak type; If it managed memory leak or You don't understand it type - go to 2 point) Or go to 3 point if it Unmanaged memory leak;

2. Analyze Managed Memory Use MS Visual Studio (if You have professional or enterprise version). Debug managed memory.
   You can find which objects uses the most managed memory.
   If You don't have MS Visual Studio (Pro or Enterprise) - try to use Azure Analyze Report, Use standart windows dump Analyze, DebugDiag2 or use WinDbg; All WinDbg info in the internet is about Managed Memory analyze. For Example: https://stackify.com/using-windbg-to-analyze-net-crash-dumps-async-crash/

3. Analyze Unmanaged Memory It is really difficult to find any info about this memory leak type. WinDbg can help us here too. You can read really cool article about it here: http://kate-butenko.blogspot.ru/2012/07/investigating-issues-with-unmanaged.html

My WinDbg help file

• Set symbols: .sympath SRV*c:\localsymbols*http://msdl.microsoft.com/download/symbols 

• Load netext lib: .load netext

• Index dump file: !windex

• Get netext command list, offcourse use Your path: .cmdtree C:\Program Files (x86)\Windows Kits\8.1\Debuggers\x64\netext.tl

• Get heap info, all objects >2Mb: !dumpheap -stat -min 2000

• Get heap info, all objects >2Mb, write results in file: .logopen C:\DumpAnalyse\crash.log; !dumpheap -stat -min 2000; .logclose;

• Fix problems if You have them with symbols load: .symfix+ C:\localsymbols .reload

• Get Objects info: !address -summary

• Get GC info (managed memory analyze): !eeheap -gc

P.S.

And for desert: WHAT PROBLEM we had in our App?)

1. It was unmanaged and managed memory problem simultaniously;
2. In managed memory we just had a lot of objects of one type. We have used SignalR in our project, so it was SignalR objects in top of high level cashe objects (that GC doesn't clean). Offcourse, a lot of time we thinked that It is SignalR problem, internal or in our code.
   But fixing all issues with SignalR use doesn't help (App crashes happen rarely but it does happen). Sometimes application just overflow memory and that's all. So after hours of research I've stumbled on article about unmanaged memory, after next few hours I've found Kates article.
3. So then I have found that we also had heap memory overflow. So When I dip into heap using WinDbg, I have found chars like that: "S.y.s.t.e.m..T.h.r.e.a.d.i.n.g..E.x.c.e.p.t.i.o.n". Just standart exception text string. MANY MANY exception thread strings.
   We use log4net logging for server exceptions. So, the problem was that signalR in some cases cyclically spams server with 500, so log4net trying to write this stackTrace into log and overflows server; + stackTraces keeps managed memory objects, so You have problems in both memory types. Fixing that cases kick off memory leaks. So I sleep very well now)

I hope that this info was usefull for You. No leaks, no headache, sleep well, repeat)