CODEWITHSUNDEEP
asp.netsqliis-6
AaronJAnderson
AaronJAnderson

Reputation: 1724

ASP.NET Requests Queued causes website to crumble. SQL backend, IIS6

I have inherited a somewhat complex system (and problem) that I need help with.

I have a webserver w/ the following specs:

This system serves a large number of people and bandwidth is fairly constant during business hours reaching ~ 68,000kbit/s

There are moments when the system "comes down" - site gets very slow which generates a lot of phone calls. Things usually slow down for 60 seconds, but has varied greatly in length. Sometimes only a few seconds and sometimes 3 minutes or more.

I have my app pools set to recycle somewhere about 600mb of consumed memory. That's not exact but they recycle on their own with much success. At times I recycle the "main" pool manually to clear the problem I'm describing.

This is what I know is going on when things are running slow.

In my mind here is what I imagine is happening.

Someone asks the system to generate a report or glob of data. This spins up a process that consumes a large number of threads (ie, all available threads) This causes all other requests to the system to wait in the ASP.NET que pool which essentially kills the site. The lack of activity causes the network traffic to dip.

I have read many articles about thread queues, thread pools, etc. This is a good example: http://williablog.net/williablog/post/2008/12/02/Increase-ASPNET-Scalability-Instantly.aspx and does what I believe is a clue to help me solve my problem... but I'm not sure. My "Machine.config" file for the version of asp.net that I am using does not specify any of the thread values listed in the article so we are default for everything which I believe is incorrect given our situation.

If you were me; What would you do next? Where do you think the problem is?

edit: Here is a screenshot. It should be obvious when the problem is happening. https://i.sstatic.net/1St0T.png

edit:

I want to change these values for our setup. A few questions first:

1) After making the changes, what needs to be restarted for them to take effect?

2) How do these settings look for a system with 8 physical cores?

maxconnection = 96
maxIoThreads = 100
maxWorkerThreads = 100
minFreeThreads = 704 
minLocalRequestFreeThreads = 608

Upvotes: 4

Views: 2965

Answers (2)

Zach Bonham
Zach Bonham

Reputation: 6817

Not fun.

Many root causes share common symptoms which makes it difficult to diagnose without getting dirty with the application. :) Pardon if some of these steps were implied.

Some next steps might be:

  • Review the IIS logs of each site looking for things like:
    • HTTP response codes (5xx,4xx,3xx)
    • Request response times
  • Review Windows Event Logs
    • How often are application pools cycling?
    • Application errors, etc.
  • Verify processModel settings as suggested by @vinayc to make sure predecessor didn't get 'tricky'
  • Install DebugDiag, its a surprisingly good tool for some basic analysis of memory and crash related problems.
    • This can also help you capture memory snaps to diagnose later.
    • Tess Ferrandez blog can help make heads/tails of memory snap analysis.
  • Understand how many web applications are running in each AppPool.
  • Investigate using a 'web garden' to possibly help minimize number of users impacted by 'slow down'
  • Is a virus scanner enabled? Is it running? If so, verify exclusions.
  • Are application teams available to help troubleshoot? Identify if they have any custom application instrumentation that might help diagnose problem.

Is the behavior 'new'? Or has it always been there? If 'new', can you track down which deployment might have caused the new behavior?

Could the the description given of the 'slow down' behavior be attributed to an apppool recycle and resulting jitting of the application again? ala - the first request syndrome.

Reviewing the logs helps understand how the sites/applications are being used, which can be especially important if you don't own the codebase. Logparser is an excellent tool for doing some IIS log analysis (as well as other formats).

Good luck!

Z

Upvotes: 2

VinayC
VinayC

Reputation: 49185

The settings that your are talking are part of processModel element under system.web element from machine.config. For IIS6, following are applicable:

autoConfig
maxIoThreads
maxWorkerThreads
minIoThreads
minWorkerThreads
requestQueueLimit
responseDeadlockInterval

Typically, you will only find autoConfig="true" and not other elements. Auto-config sets the values as per your machine configuration - the tuning is done as per recommended values (see Threading Explained section from this article) which are same as sighted by the link that you have provided.
The article although dated, i excellent resource if you want to tune up these settings manually.

On the other hand, at the load that you are serving, I would recommend two things (if you haven't tried already)

  • Use output caching aggressively - even if the data is dynamic, caching for say 30-60 seconds can give a definite boost at your load
  • If you suspect certain requests are hogging too many threads then attempt to move those resources under different app-pool (you can use different web-site with different sub-domain or you can use different virtual directory/application and choose different app-pool)

Upvotes: 1

Related Questions