Unable to configure IIS/Asp.NET to process many asynchronous requests concurrently

Question

I am trying to explore the asynchronous ASP.NET requests. The Asp.NET application is hosted by IIS 8.

The client side is issuing many POST requests using the following code:

private static async Task<Result> IOAsync(Uri url, byte[] body)
{
    var webClient = new WebClient();
    webClient.Headers["Content-Type"] = "application/json";
    return DeserializeFromBytes(await webClient.UploadDataTaskAsync(url, "POST", body));
}

private static Result DeserializeFromBytes(byte[] bytes)
{
    using (var jsonTextReader = new JsonTextReader(new StreamReader(new MemoryStream(bytes))))
    {
        return new JsonSerializer().Deserialize<Result>(jsonTextReader);
    }
}

The Asp.NET code on the other side is the Asp.NET Web application created by the VS 2013 New Project Wizard with a slight modification in the HomeController:

public class HomeController : Controller
{
    public ActionResult Index()
    {
        return View();
    }

    public ActionResult About()
    {
        ViewBag.Message = "Your application description page.";

        return View();
    }

    public ActionResult Contact()
    {
        ViewBag.Message = "Your contact page.";

        return View();
    }

    [System.Web.Mvc.HttpPost]
    public async Task<JsonResult> SampleAPIAsync([FromBody] BaseContext c, Runner.BasicIOStrategy strategy)
    {
        return Json(await ((IAsyncRunner)Runner.Create((Runner.IOStrategy)strategy)).IOAsync(c));
    }
}

The SampleAPIAsync is meant to explore different approaches to do database IO, of which only one is truly asynchronous, the rest are there to demonstrate the usual misconceptions about "simulating" it using Task.Run and similar.

In my particular scenario the IOAsync method is truly asynchronous:

private async Task<Result> IOAsync(string sqlPauseDuration)
{
    Result result;
    using (var conn = new SqlConnection(m_connectionString))
    using (var cmd = CreateCommand(conn, sqlPauseDuration))
    {
        await conn.OpenAsync();
        using (var reader = await cmd.ExecuteReaderAsync())
        {
            await reader.ReadAsync();
            result = new Result(reader.GetDateTime(0), reader.GetGuid(1));
        }
    }
    return result;
}

private SqlCommand CreateCommand(SqlConnection conn, string sqlPauseDuration)
{
    const string SQL = "WAITFOR DELAY @Duration;SELECT GETDATE(),NEWID()";
    var sqlCommand = new SqlCommand(SQL, conn) { CommandTimeout = QueryTimeout };
    sqlCommand.Parameters.Add(new SqlParameter("Duration", sqlPauseDuration));
    return sqlCommand;
}

So, as you can see, everything is asynchronous. I am using the Performance Monitor to check the Thread Count on:

1. The client application
2. The IIS worker process (w3wp.exe)
3. The Sql Server process

My expectation is to see a relatively flat line for the client and the IIS worker process and a sudden spike in the Sql Server.

This never happens. Doing 600 requests ultimately translating to running

WAITFOR DELAY @Duration;SELECT GETDATE(),NEWID()

on the sql server with the @Duration of 20 seconds does not generate any spikes and takes 5 minutes (almost exactly)! From which I conclude the requests are not being processed with the sufficient concurrency. If I am to guess, I would say it processes (5 * 60) / 20 = 15 requests concurrently.

Note, that:

• Before doing the test I have "cold run" requests to warm up the IIS and the Sql Server. So, the startup time does not contribute to the equation.
• When I run the IOAsync directly from the client using the same setup - 600 requests and 20 seconds I do see the expected spike of the Thread Count on the Sql Server and all of the 600 requests complete at the same time after less than 21 second!

From which I conclude the problem is on the Asp.NET/IIS side.

Googling the issue prompted me to change the files C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Aspnet.config and c:\Windows\Microsoft.NET\Framework\v4.0.30319\Aspnet.config like this:

<?xml version="1.0" encoding="UTF-8" ?>
<configuration>
    <runtime>
        <legacyUnhandledExceptionPolicy enabled="false" />
        <legacyImpersonationPolicy enabled="true"/>
        <alwaysFlowImpersonationPolicy enabled="false"/>
        <SymbolReadingPolicy enabled="1" />
        <shadowCopyVerifyByTimestamp enabled="true"/>
    </runtime>
    <startup useLegacyV2RuntimeActivationPolicy="true" />
    <system.web> 
        <applicationPool maxConcurrentRequestsPerCPU="5000" maxConcurrentThreadsPerCPU="0" requestQueueLimit="5000"/> 
    </system.web>
</configuration>

Notice the /configuration/system.web/applicationPool element. Doing so does not affect the outcome - still apparent 15 requests at the same time.

What am I doing wrong?

EDIT 1

Not sure if it is relevant, this is how a typical HTTP request and response look like when observed through Fiddler:

Request

POST http://canws212:12345/Home/SampleAPIAsync?strategy=Async HTTP/1.1
Content-Type: application/json
Host: canws212:12345
Content-Length: 174
Expect: 100-continue

{"IsBlocking":false,"Strategy":0,"Server":"CANWS212","Count":600,"QueryTimeout":30,"DurationSeconds":20,"ConnectionTimeout":4,"DBServer":"localhost","DBName":"tip_DFControl"}

Response

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/8.0
X-AspNetMvc-Version: 5.2
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 19 Nov 2015 17:37:15 GMT
Content-Length: 83

{"Id":"16e8c3a2-fc95-446a-9459-7a89f368e074","Timestamp":"\/Date(1447954635240)\/"}

EDIT 2

Please, find below the web.config of the Asp.Net application:

<?xml version="1.0"?>
<!--
  For more information on how to configure your ASP.NET application, please visit
  http://go.microsoft.com/fwlink/?LinkId=301880
  -->
<configuration>
  <appSettings>
    <add key="webpages:Version" value="3.0.0.0"/>
    <add key="webpages:Enabled" value="false"/>
    <add key="ClientValidationEnabled" value="true"/>
    <add key="UnobtrusiveJavaScriptEnabled" value="true"/>
  </appSettings>
  <!--
    For a description of web.config changes see http://go.microsoft.com/fwlink/?LinkId=235367.

    The following attributes can be set on the <httpRuntime> tag.
      <system.Web>
        <httpRuntime targetFramework="4.5.2" />
      </system.Web>
  -->
  <system.web>
    <compilation debug="true" targetFramework="4.5.2"/>
    <httpRuntime targetFramework="4.5"/>
  </system.web>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" culture="neutral" publicKeyToken="30ad4fe6b2a6aeed"/>
        <bindingRedirect oldVersion="0.0.0.0-6.0.0.0" newVersion="6.0.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Optimization" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="1.1.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="WebGrease" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="0.0.0.0-1.5.2.14234" newVersion="1.5.2.14234"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Helpers" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-3.0.0.0" newVersion="3.0.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Web.WebPages" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-3.0.0.0" newVersion="3.0.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Mvc" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-5.2.3.0" newVersion="5.2.3.0"/>
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

EDIT 3

Turned off the sessionState in the web.config:

<system.web>
  <compilation debug="true" targetFramework="4.5.2"/>
  <httpRuntime targetFramework="4.5"/>
  <sessionState mode="Off" />
</system.web>

No loving, same result.

EDIT 4

Checked the <limits> element, it is the default:

PS C:\> &"C:\Program Files\IIS Express\appcmd.exe" list config -section:system.applicationHost/sites |sls limits | group

Count Name                      Group
----- ----                      -----
   30       <limits />          {      <limits />,       <limits />,       <limits />,       <limits />...}


PS C:\>

So, I suppose I am not limiting it artificially. The InetMgr reports the following:

On the other hand I am running Windows 8 with IIS 8, so http://weblogs.asp.net/owscott/windows-8-iis-8-concurrent-requests-limit should apply. I tried to change the limit to 5000 both as site defaults and on my site, but it did not bear any fruits - same result.

What I did was:

1. Change the limit to 5000 on the relevant site using InetMgr GUI.
2. iisreset
3. Run my test

Same result.

EDIT 5

My JSON body deserialization code has a bug - it used the default value for the duration, rather than the one in the request body. So, it was not 20 seconds, but 5 seconds duration per SQL statement. And the formula I used is wrong, it should be

NumberOfConcurrentRequests = TotalRequests / NumberOfBatches 
                           = TotalRequests / (TotalTime / OneRequestDuration)
                           = (TotalRequests / TotalTime) * OneRequestDuration
                           = (600 / 300) * 5
                           = 10

Which is consistent with http://weblogs.asp.net/owscott/windows-8-iis-8-concurrent-requests-limit

Now, I have deployed to a Windows Server 2012 and there I am able to issue 75 DB requests, each 10 seconds long which complete in almost exactly 10 seconds all together. But not 76, from which I conclude the actual concurrency limit is 75. Not 5000. Still looking for the clues.

EDIT 6

Following the suggestion of Stephen Cleary I have replaced all the DB IO with Task.Delay and stopped the Sql Server.

Without asp.net I can easy run 600 Task.Delay of 10 seconds and all of them end in 10 seconds (with a little tiny extra).

With asp.net the result is consistent - 75 requests are fully concurrent and asynchronous. Above that, the picture is different. So, 80 requests took 16 seconds, 100 requests took 20 seconds and 200 took 30 seconds. It is obvious to me that the requests are throttled either by Asp.NET or IIS, just as before when the DB IO was exercised.

Answer 1

If you're running IIS 8.0 on Windows 8 rather than Server 2012, there are hard-coded limits on some things. You should be able to increase it above the default (the default limit would be 10 for Windows 8 Professional, 3 for Basic edition): Default Limits for Web Sites , also Windows 8 / IIS 8 Concurrent Requests Limit‌​.

So, in the <system.applicationHost> section of applicationHost.config (normally located in C:\Windows\System32\inetsrv\config) you can add a <limits> element as shown near the end of this:

<sites>
    <site name="Default Web Site" id="1" serverAutoStart="true">
        <application path="/">
            <virtualDirectory path="/" physicalPath="C:\inetpub\wwwroot" />
        </application>
        <bindings>
            <binding protocol="http" bindingInformation="*:80:" />
        </bindings>
    </site>
     <site name="X" id="2">
        <application path="/" applicationPool="X">
            <virtualDirectory path="/" physicalPath="C:\Inetpub\wwwroot\X" />
        </application>
        <bindings>
            <binding protocol="http" bindingInformation="*:80:X" />
        </bindings>
        <traceFailedRequestsLogging enabled="false" />
        <limits maxConnections="40" />
    </site>
    <siteDefaults>
        <logFile logFormat="W3C" directory="%SystemDrive%\inetpub\logs\LogFiles" />
        <traceFailedRequestsLogging directory="%SystemDrive%\inetpub\logs\FailedReqLogFiles" />
    </siteDefaults>
    <applicationDefaults applicationPool="DefaultAppPool" />
    <virtualDirectoryDefaults allowSubDirConfig="true" />
</sites>

With a maximum usable value of 40.

It could be safer to use IIS Manager to select the web site then change the value in "Advanced settings..."->"Limits"->"Maximum Concurrent Connections".

If you have a need for more concurrent connections, one option would be to get a trial version of Windows Server 2012 and run it in a virtual machine (Hyper-V). I suggest getting the server version which corresponds to the development platform, e.g. Server 2012 R2 to go with Windows 8.1, so that you find the same problems on both.

Answer 2

At the end of the day and after following the given advices I have found the culprit. Surprisingly it was Fiddler who was throttling the requests! I suppose there is a Fiddler setting somewhere controlling it.

I have discovered it by checking the ClientConnected vs ClientBeginRequest timestamps in the request statistics. While ClientConnected was the same - the time when the client sent the request, it had different ClientBeginRequest - after certain number of requests it starts to be behind with the lag progressively increasing.

Answer 3

I'm guessing that this is due to the client-side "max two concurrent connections per endpoint"-rule as mandated by the original HTTP specification.

You'll need to set ServicePointManager.DefaultConnectionLimit to an appropriate value to allow more concurrent connections.

For example: (do this on the client-side before sending requests)

ServicePointManager.DefaultConnectionLimit = int.MaxValue;

This will allow your application to open an unlimited (in practice) number of concurrent HTTP connections.

---

The following quote is from section 8.1.4 of the HTTP/1.1 RFC:

A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy [...] These guidelines are intended to improve HTTP response times and avoid congestion.

Therefore, a compliant client would typically have to ensure that it never has more than two open connections to a single server.

However, this have changed and you're allowed to override this behavior. The following quote is from section 6.4 of RFC 7230 which is an update of the HTTP/1.1 protocol:

Previous revisions of HTTP gave a specific number of connections as a ceiling, but this was found to be impractical for many applications. As a result, this specification does not mandate a particular maximum number of connections but, instead, encourages clients to be conservative when opening multiple connections.