How do you profile a production ASP.NET applicaiton?

Question

We have some performance problems with one of our applications. I thought about using something like dotTrace to find out where the problems are, but dotTrace would probably degrade performance even more.

What's the best way to profile an application that's on a prod environment w/o affecting performance too much?

Answer 1

Profiling memory or cpu?

Memory: the best way would be to create a memory dump of the w3wp process (launch task manager, right click the process then "create dump"), then copy the dump to your local machine and analyse it with WinDbg. And look at which classes consume the most memory. There are lots of questions/answers here on Stackoverflow on how do do that (how to use WinDbg and analyse the .NET heap).

CPU: we use a short command-line profiler by Sam Saffron (woohoo, one of the creators of Stackoverflow!) His project is abandoned, but we forked it and maintain it here. https://github.com/jitbit/cpu-analyzer Everyone's welcome to contribute. It attaches to your threads using Microsoft's DbgManager and finds call-stacks that take longest time to execute.

Answer 2

I would recommend to take several memory dumps of the process in Production, look at all the stack traces and see if you find a pattern.

Answer 3

You can use ants profile as well as performance counter of the system. It will help you to determine whats the problem.

Here are some details about performance counter..

http://msdn.microsoft.com/en-us/library/fxk122b4.aspx

http://msdn.microsoft.com/en-us/library/ms979204.aspx

http://www.codeproject.com/KB/dotnet/perfcounter.aspx

Answer 4

You could use ants profiler

http://www.red-gate.com/products/ants_performance_profiler/index2.htm

They claim that "the overhead was hardly noticeable".

There is a 14 day free trial so you could give it a try.

Edit: I agree with John's comment, it will disrupt, require some down time, to get it started / stopped. Best to use it on a test environment to identify the bottle necks.

Answer 5

Did you load-test the application on a number of sessions that's anywhere near the actual load of the production environment?

The first thing that comes to mind is that your app is not scaling well under load or that your db is not scaling well with an increase in size (causing this way problems even with a very limited number of concurrent sessions) but it could be anything really.

My suggestion is to replicate the production environment and run proper load-testing then look at the data and it'll give you some clue.

you don't wanna play games with your production environment, but if you don't have it already you could use logging to keep track of the sequence and time spans of key-events and take it from there.

Answer 6

The general answer is "don't do it".

Other than that, you can gain a lot by using performance counters. If the built-in counters don't help, you can create your own.

Among other things, the performance counters may give you an idea of how to reproduce the performance problems through load testing.

The next idea is to narrow down the area you're interested in. There's no sense impacting performance for the entire application if it turns out to be your web service access that's slow.

Next, be sure to have instrumented your application, preferably by using configuration. The Enterprise Library Logging Application Block is great for that, as it allows you to add the logging to your application, but have it configured off. Then, you can configure what kind of information to log, and where to log it to.

This gives you choices about how expensive the logging should be, from logging to the event log to logging to an XML file. And you can decided this all at runtime.

Finally, you're not going to be able to use dotTrace or something else that requires restarting IIS an adding code to your running application. Not in production. The ideas above are for the purpose of not needing to do so.