CODEWITHSUNDEEP
c#iisiis-7cpu-usage
EboMike
EboMike

Reputation: 77762

CPU on IIS server pegged at 100% in BufferAllocator.GetBuffer()

I have an IIS7 application pool that handles a large number of RESTful requests. When things get heated, the CPU usage hits 100% and the requests take many seconds to be processed.

When profiling with ANTS, we found that very often most of the CPU time goes here:

System.Web.Hosting.PipelineRuntime.ProcessRequestNotification
 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper
  (Unmanaged code)
   System.Web.Hosting.PipelineRuntime.ProcessRequestNotification
    System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper
     System.Web.HttpRuntime.ProcessRequestNotificationPrivate
      System.Web.HttpApplication.BeginProcessRequestNotification
       System.Web.HttpApplication+PipelineStepManager.ResumeSteps
        System.Web.HttpApplication.ExecuteStep
         System.Web.HttpApplication+CallFilterExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute
          System.Web.HttpResponse.FilterOutput
           System.Web.HttpWriter.FilterIntegrated
            System.Web.Hosting.IIS7WorkerRequest.GetBufferedResponseChunks
             System.Web.Hosting.RecyclableArrayHelper.GetIntegerArray
              >> System.Web.BufferAllocator.GetBuffer
               (Thread blocked)
              >> System.Web.BufferAllocator.ReuseBuffer
               (Thread blocked)

There are actually several different stack traces, but they all invariably end in GetBuffer() or ReuseBuffer().

Both GetBuffer() and ReuseBuffer() begin with a lock(), so I figure the CPU spends a lot of time in a spinlock (my understanding is that lock spins for a bit before putting the thread to sleep).

My question - is this a common place for the CPU to be spending its time on? This is all entirely in IIS code, so what can I do to reduce the CPU load? Is that a configuration issue, or is this the result of actions that our application did earlier?

The machines are pretty beefy, they have 4 quadcores. I don't have the number of threads running currently available.

Upvotes: 2

Views: 714

Answers (1)

EboMike
EboMike

Reputation: 77762

It was pretty much what we suspected - the threads were all spending a stupid amount of time in the spinlock, since they all compete for the same lock. We had hundreds of threads.

The fix was to have more processes with fewer threads - now the CPU usage is reasonable.

Upvotes: 3

Related Questions