CODEWITHSUNDEEP
asp.net-mvcazureiisazure-cloud-services
Blisco
Blisco

Reputation: 620

Application_Start loops when Azure web role starts up

We have an ASP.NET MVC application running in an Azure cloud service web role. There is a lot of startup code in MvcApplication.Application_Start(), doing database initialization among other things, and this can take a couple of minutes or more to run.

Sometimes, when a role instance starts up (either on deployment or following a restart), the application fails to start up first time. We can see from the logs that Application_Start fires, gets a certain way through, and then starts again from the beginning. After repeating this process a few times, the application does usually complete its startup routine and begin handling requests, but occasionally it gets into a seemingly never-ending loop, for which the only solution is to reboot or reimage the role instance.

We are not seeing any errors being logged, and Application_End is not firing at all. Serilog's HttpRequestIdEnricher is logging a different request ID each time Application_Start fires, indicating that there is a fresh HttpContext.Current each time. I have never seen this behaviour when running locally, only in Azure.

Any ideas what is going on?

Update

This behaviour seems to happen only when multiple requests are being fired at the website. It most commonly (and infuriatingly) happens when a role instance in production is restarted; the UAT environment normally starts up fine, but I can provoke this issue by making multiple requests in different browser tabs. I also tried implementing something like this - making WebRole.OnStart ping the web URL to ensure the site was running before exiting - but that only ensured that the site never came up.

So my guess is that IIS is to blame: when a new request comes in while Application_Start is in progress, it seems to be restarting the application rather than letting the original request complete.

Upvotes: 3

Views: 537

Answers (1)

Blisco
Blisco

Reputation: 620

I seem to have tracked this down to one or two settings in the IIS app pool configuration.

IIS app pool ping settings

I initially assumed Startup Time Limit was the most likely culprit, and tried setting that to 240 seconds (since our application typically takes 2-3 minutes to start up), but the problem persisted. I subsequently also changed Ping Maximum Response Time to 240 seconds, and that seems to have cured it.

When the startup got stuck in a loop there was usually (not always) a 2-minute interval between successive starts, so I assume IIS was waiting 30 seconds (the default Ping Period), then firing a request and waiting 90 seconds for a response, before restarting the worker process.

This Powershell code, in a script that runs when the web role starts up, configures the relevant settings:

Import-Module WebAdministration

Get-ChildItem IIS:\AppPools | ForEach-Object {
    $appPool = "IIS:\AppPools\" + $_.Name
    $startupLimit = New-TimeSpan -Minutes 4
    Set-ItemProperty $appPool -Name processModel.startupTimeLimit -Value $startupLimit
    Set-ItemProperty $appPool -Name processModel.pingResponseTime -Value $startupLimit
}

It should be possible to do the same thing with Microsoft.Web.Administration.ServerManager, but I haven't tried.

Upvotes: 2

Related Questions