CODEWITHSUNDEEP
asp.netperformancecrash-dumps
wasigh
wasigh

Reputation: 905

Asp.net application slow but CPU is at 40% max

I have a strange situation on a production server. Connection for asp.net get queued but the CPU is only at 40%. Also the database runs fine at 30% CPU.

Some more history as requested in the comments:

My conclusion is that something else is stopping the server from handling the requests faster. Possible suspects

To find out what the proces is doing I created to minidumps.

I managed to create two MemoryDumps 20 seconds apart. This is the output of the first:

!threadpool
CPU utilization 6%
Worker Thread: Total: 95 Running: 72 Idle: 23 MaxLimit: 200 MinLimit: 100
Work Request in Queue: 1
--------------------------------------
Number of Timers: 64

and the output of the second:

!threadpool
CPU utilization 9%
Worker Thread: Total: 111 Running: 111 Idle: 0 MaxLimit: 200 MinLimit: 100
Work Request in Queue: 1589

As you can see there are a lot of Request in Queue.

Question 1: what does it mean that there are 1589 requests in queue. Does it mean something is blocking?

The !threadpool list contains mostly these entries: Unknown Function: 6a2aa293 Context: 01cd1558 AsyncTimerCallbackCompletion TimerInfo@023a2cb0

If I you into depth with the AsyncTimerCallbackCompletion

!dumpheap -type TimerCallback

Then I look at the objects in the TimerCallback and most of them are of types:

System.Web.SessionState.SessionStateModule
System.Web.Caching.CacheCommon

Question 2: Does it make any sense that those Objects hava a timer, and so much? Should I prevent this. And how?

Main Question do I miss any obvious problems why I'm queueing connections and not maxing out the CPU?

I succeeded in making a crashdump during a peak. Analyzing it with debugdiag gave me this warning:

Detected possible blocking or leaked critical section at webengine!g_AppDomainLock owned by thread 65 in Hang Dump.dmp
Impact of this lock
25.00% of threads blocked
(Threads 11 20 29 30 31 32 33 39 40 41 42 74 75 76 77 78 79 80 81 82 83)

The following functions are trying to enter this critical section
webengine!GetAppDomain+c9

The following module(s) are involved with this critical section
\\?\C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\webengine.dll from Microsoft Corporation

A quick google search doesn't give me any results. Does somebody has a clue?

Upvotes: 15

Views: 9947

Answers (5)

Lightsauce
Lightsauce

Reputation: 41

Was anybody able to confirm this worked for them? I've found that answer across the web, and there are zero confirmations that the posted answer fixed this problem for them. With that being said, I don't really give it credibility as the answer is provided by the question poster.

I got the same problem recently:

Detected possible blocking or leaked critical section at webengine!g_AppDomainLock owned by thread 16 in w3wp.exe__DefaultAppPool__PID__3920__Date__04_26_2011__Time_10_40_42AM__109__IIS_COM+ Hang Dump.dmp Impact of this lock

4.17% of threads blocked (Threads 17) The following functions are trying to enter this critical section webengine!GetAppDomain+c9 The following module(s) are involved with this critical section \?\c:\WINDOWS\microsoft.net\framework\v2.0.50727\webengine.dll from Microsoft Corporation

This is the recommendation posted by Microsoft to further troubleshoot:

The following vendors were identified for follow up based on root cause analysis Microsoft Corporation Please follow up with the vendors identified above. Consider the following approach to determine root cause for this critical section problem:

  1. Enable 'lock checks' in Application Verifier A. Download Application Verifier from the following URL: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=c4a25ab9-649d-4a1b-b4a7-c9d8b095df18&displaylang=en B. Enable 'lock checks' for this process by running the following command:

    Appverif.exe -enable locks -for w3wp.exe C. See the following document for more information on Application Verifier: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnappcom/html/appverifier.asp?frame=true

  2. Use a DebugDiag crash rule to monitor the application for exceptions

Upvotes: 0

Tim P.
Tim P.

Reputation: 2942

I know this is an old thread but it's one of the first Google hits for people with poor ASP.NET site performance. So I will throw out a few recommendations:

1) Asynchronous Programming will solve the root cause. While you're calling out to a webservice to do your actual business logic, those request threads are just sitting there waiting on the response. They could be used instead to service another incoming request. This will reduce your Queue Length dramatically if not eliminate it entirely. Asynchronous programming is about scalability, not individual request performance. This is achieved quite easy in .NET 4.5 with the Async/Await pattern. ASP.NET injects threads at a rate of 2 per minute, so unless you are re-using those existing threads, you're going to quickly run out with the site load you are receiving. In addition, spinning up more threads is a small performance hit; it takes up more RAM and time to allocate that RAM. Just increasing the thread pool size in the machine.config won't fix the underlying problem. Unless you add more CPUs, adding more threads won't really help since it's still a misallocation of resources and you can also context-switch yourself to death by having too many threads and too little CPU.

2) From a popular article on threading in IIS 7.5: If your ASP.NET application is using web services (WFC or ASMX) or System.Net to communicate with a backend over HTTP you may need to increase connectionManagement/maxconnection. For ASP.NET applications, this is limited to 12 * #CPUs by the autoConfig feature. This means that on a quad-proc, you can have at most 12 * 4 = 48 concurrent connections to an IP end point. Because this is tied to autoConfig, the easiest way to increase maxconnection in an ASP.NET application is to set System.Net.ServicePointManager.DefaultConnectionLimit programatically, from Application_Start, for example. Set the value to the number of concurrent System.Net connections you expect your application to use. I've set this to Int32.MaxValue and not had any side effects, so you might try that--this is actually the default used in the native HTTP stack, WinHTTP. If you're not able to set System.Net.ServicePointManager.DefaultConnectionLimit programmatically, you'll need to disable autoConfig , but that means you also need to set maxWorkerThreads and maxIoThreads. You won't need to set minFreeThreads or minLocalRequestFreeThreads if you're not using classic/ISAPI mode.

3) You should really look at load-balancing if you're getting 20k unique visitors per hour. If every user did 10-20 AJAX requests per hour, you're easily talking about 1 million or more web service calls to your backend. Throwing up another server would reduce the load on the primary server. Combining this with async/await, and you've put yourself in a good situation where you can easily throw hardware at the problem (scaling out). There are multiple benefits here such as hardware redundancy, geolocation, and also performance. If you're using a cloud provider such as AWS or RackSpace, spinning up another VM with your app on it is easy enough that it can be done from your mobile phone. Cloud computing is too cheap nowadays to even have a queue length at all. You could do this to provide the performance benefits even before you make the switch to an asynchronous programming model.

4) Scaling Up: adding more hardware to your server(s) help because it providers better stability when you have additional threads. More threads means you need more CPUs and RAM. And even after you've gotten async/await under your belt, you'll still want to fine-tune those web service requests if you can. This could mean adding in a caching layer or beefing up your database system. You do NOT want to maximize the CPU on that single server. Once the CPU reaches 80%, ASP.NET will stop injecting more threads into the system. It doesn't matter if the worker process is sitting at 0%, if the overall system CPU utilization as reported by Task Manager reaches 80%, then thread injection stops and requests begin to queue. Weird things with garbage collection also happens when it detects a high CPU load on the server.

Upvotes: 2

wasigh
wasigh

Reputation: 905

The worker processes handling the queue was the real dealbreaker. Probably connected with the website calling webservices on the same host. Thus creating a kind of deadlock.

I changed the machine.config to to following:

<processModel
        autoConfig="false"
        maxWorkerThreads="100"
        maxIoThreads="100"
        minWorkerThreads="50"
        minIoThreads="50" />

Standard this processModel is set to autoConfig="true"

With the new config the webserver is handling the requests fast enough to not get queued.

Upvotes: 4

ChrisLively
ChrisLively

Reputation: 88092

I'm with realworldcoder: IIS works by having Worker Processes handle the incoming requests. If the requests get stacked up, as it appears is happening, then performance takes a nose dive.

There are several possible things to do/check for.

  1. Fire up Activity Monitor on the SQL Server. You want to see what queries are taking the longest and, depending on the results, make changes to reduce their execution time. Long queries can cause the thread the page is executing under to block, reducing the number of connections you can support.

  2. Look at the number of queries, and the time they take to execute, for these page/ajax calls. I've seen pages with dozens of unnecessary queries that get executed for an Ajax call simply because .Net executes the entire page cycle even when only a particular method needed to be run. You might split those calls into regular web handlers (.ashx) pages that way you can better control exactly what happens.

  3. Consider increasing the number of worker processes IIS has to handle incoming requests. The default for a new app pool is 1 process with 20 threads. This is usually enough to handle tons of requests; however, if the requests are blocking due to waiting on the DB server or some other resource it can cause the pipeline to stack up. Bear in mind that this can have either a positive or negative impact to both performance and regular functioning of your application. So do some research then test, test, test.

  4. Consider reducing or eliminating your usage of session. Either way, look at the memory usage of it, potentially add more ram to your web server. Session data is serialized and deserialized for every page load (including ajax calls) regardless of whether the data is used or not. depending on what you are storing in session it can have a serious negative impact on your site. If you aren't using it, then make sure it's completely turned off in your web.config. Note that these issues only get worse if you store session off of the web server as you then become bound to the speed of the network when a page retrieves and stores it.

  5. Look at the sites performance counters around JIT (Just-In-Time) compiling. This should be nearly non-existent. I've seen sites brought to their knees by massive amounts of JIT. Once those pages were recoded to eliminate it, the sites started flying again.

  6. Look at different caching strategies (I don't consider session a real caching solution). Perhaps there are things that you constantly request that you don't really need to constantly pull out of the DB server. A friend of mine has a site where they cache entire web pages as physical files for dynamic content, including their discussion groups. This has radically increased their performance; but it is a major architectural change.

The above are just a couple things to look at. You basically need to get further into the details to find out exactly what is going on and most of the regular performance counters aren't going to give you that clarity.

Upvotes: 3

realworldcoder
realworldcoder

Reputation: 699

Too many ASP.NET queued requests will destroy performance. There are a very limited number of request threads.

Try to free up those threads by processing slow parts of your pages asynchronously or do anything else you can to bring down page execution times.

Upvotes: 2

Related Questions