high cpu in IIS

Question

I'm developing a POS application that has a local database on each POS computer, and communicates with the server using WCF hosted in IIS. The application has been deployed in several customers for over a year now.

About a week ago, we've started getting reports from one of our customers that the server that the IIS is hosted on is very slow. When I've checked the issue, I saw the application pool with my process rocket to almost 100% cpu on an 8 cpu server.

I've checked the SQL Activity Monitor and network volume, and they showed no significant overload beyond what we usually see.

When checking the threads in Process Explorer, I saw lots of threads repeatedly calling CreateApplicationContext. I've tried installing .Net 2.0 SP1, according to some posts I found on the net, but it didn't solve the problem and replaced the function calls with CLRCreateManagedInstance.

I'm about to capture a dump using adplus and windbg of the IIS processes and try to figure out what's wrong.

Has anyone encountered something like this or has an idea which directory I should check ?

p.s. The same version of the application is deployed in another customer, and there it works just fine. I also tried rolling back versions (even very old versions) and it still behaves exactly the same.

Edit: well, problem solved, turns out I've had an SQL query in there that didn't limit the result set, and when the customer went over a certain number of rows, it started bogging down the server. Took me two days to find it, because of all the surrounding noise in the logs, but I waited for the night and took a dump then, which immediately showed me the query.

Answer 1

We had the same problem. The CPU usage of some IIS application pool processes was so high, that CPU usage on the Web Server was around 100%.

First we used DebugDiag and ProcMon to narrow down the problem. See here: http://www.iis.net/learn/troubleshoot/performance-issues/troubleshooting-high-cpu-in-an-iis-7x-application-pool

We found many "This thread is waiting in a WaitOne" messages in the DebugDiag analysis. This indicated that the requests are for some reasong waiting for each other. So we went about looking for shared ressources. The only one we could really find was the database. So we figured that despite the Web Server having 100% CPU usage, the real problem needed to be the database server.

My collegue investigated the case further. He did the following things:

1.) Configure parallelism of SQL Server

Using ProcMon on the Database Server he found out that there were too many locks and latches used by the SQL Server. Take a look here: http://blog.sqlauthority.com/2011/02/06/sql-server-cxpacket-parallelism-usual-solution-wait-type-day-6-of-28/

He set the number of used processors per query to 4. The default value was 0 (which I suspect equals the number of processors available - 24 in our case). You can set this using SQL Management Studio, by right-clicking on the server node itself and choosing properties.

This had a drammatical impact reducing the latches and speeding up the requests. Our guess is that SQL Server overused parallelisation of the queries which resulted in too heavy synchronization on finishing them.

2.) Creating new and missing indices in the database

We were running a third party forum software on our site which turned out to be not using any indices at all on its database. My collegue used the knowledge from here: http://www.mssqltips.com/sqlservertip/1634/using-sql-server-dmvs-to-identify-missing-indexes/ to create several new indices.

Now the case seems to be settled.

Answer 2

Usually this has nothing to do with hardware and everything to do with how IIS is configured coupled with some slightly long running queries (100+ milliseconds).

Under your application pool configuration set your web garden setting to something like 20 or more.

The web garden setting is pretty much the number of threads available to process requests for your application. If it's set to 1 then a single query could block handling other requests until it completes.

I have an app that's handling close to 3.5 million requests a day. When the web garden was set to 1, the web server CPU stayed at 100% and had a LOT of requests dropped. When I upped it to 50, the web server CPU dropped to just under 2% and no requests were dropped.

Answer 3

Don't eliminate the possibilty of a hardware issue. I had a server running slow and found it to be a motherboard issue. It was replaced under warranty.

Answer 4

Going purely on gut and doing a complete guess, it sounds like there could be sort of exception happening, the exception is getting caught at the global exception handler in the global.asax, and the exception handler is also causing throwing an exception and dumping the process. Could also be due to a virus scanning utility locking up some files. I could be WAY off, though.