IIS 8: random "(503) Server Unavailable Server Too Busy" errors

Question

I have an IIS server running Windows Server Core 2012 R2. This server will replace another server that we already have and that plays the same role. The server hosts 80 ASP.Net websites and all the sites are in the same application pool. The queue limit is at 1000 (the default).

We have homemade monitoring program that regularly makes a request to the /login.aspx page of each site sequentially to check that it's up.

On this new server, the monitoring program regularly encounters (503) Server Unavailable Server Too Busy errors.

• When the error occurs, there is nothing logged in the Event log about IIS or ASP.Net errors.
• I know that the worker process does not crash or gets recycled (w3wp keeps the same PID).
• Using Failed Request Tracing I could not find anything of value (the failing requests are logged but I can't find a cause in the log detail).
• Using Perfmon I confirmed that the number of queued request remains very low.

If I modify the monitoring program to sleep 1sec between each site, then I don't get the error anymore.

Any idea?

Answer 1

We found the cause. The difference is that our new IIS server serves content from a UNC path. The default setting that controls how ASP.NET watches files and directories is inefficient when using a UNC.

---

.Net 4.0 : registry

A while ago Microsoft released a hotfix to support another watch strategy but they did not make that the default.

Reference: https://support.microsoft.com/en-us/kb/911272. You don't need to install this hotfix on recent version of windows.

You can change the behavior for file change notification by changing the registry key HKLM\SOFTWARE\Wow6432Node\Microsoft\ASP.NET\FCNMode to 2.

.Net 4.5 : configuration setting

There is now a setting in the configuration to control the behavior for file change notification.

Reference: https://msdn.microsoft.com/en-us/library/system.web.configuration.fcnmode(v=vs.110).aspx

---

On our case we are using .Net 4.0 and setting the registry entry resolved the issue.

There is still a mystery however. Why does inefficient file change notification result in "503 errors" being sent to the clients without any trace of an issue in the web server logs? If anyone has an explanation I'd be really interested.