Live server down Sometimes asp.net application, don't find any reason why?

Question

My asp.net application is down sometimes on live server. All users face yellow error screen. When i dig in to problem i find the trace.

Exception of type 'System.Web.HttpUnhandledException' was thrown. =====>stact trace====> at System.Web.UI.Page.HandleError(Exception e) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext context) at ASP.emr_patient_callbacks_patientappointments_aspx.ProcessRequest(HttpContext context) in c:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\05f0ecab\db8ea090\App_Web_wgoawcvo.0.cs:line 0 at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

This exception occurs on randomly not on specific location or specific click. But when i restart my application on IIS the application working fine. but again some hours same problem appears.

Answer 1

As far I can see from your exception message is that could be an issue related to this view patientappointments_aspx (or a view with a similar name to this one). You'll need to check if all shutting down problems is with the same message to be sure that every time is the same problem.

If you have access to deployment environment, it may be possible to trace by Event logs.

Another suggestion to trace this erros is to have a wider try-catch combination that will trace all exceptions generated by your application (this needs to be a temporary solution just to identify the issue reason). To differentiate your treated exceptions, you could create a specific exception and those not identifiable as general exceptions.

I hope this will help to trace your problem!

Answer 2

Welcome to the fascinating world of fitting square peg in a round hole a.k.a Asynchronous Programming in ASP.NET Web Forms. Looking at the stacktrace I think you are using asp.net web forms in .NET 4.x and doing asynchronous programming somewhere in your codebase.

I've spent countless hours triaging random deadlock when I had to use async pattern in asp.net webform (.aspx/.aspx.cs). I've found sometimes the page spins forever or goes down in a tragic fashion like in your case.

Much has to do with how loosely MSFT allowed you to do async programming when it had to make it compatible with web forms. There is a right way to use Async/Await pattern in web forms and a wrong way which causes major problems.

I'd ask that you look carefully through your codebase and find where you are using Async programming. Look for keyword "async" "await", "Task" etc and see if you are using the pattern properly and not doing something that's flat out wrong such as a method like

public async void Foo(){}

You either have to do Async properly throughout the request life cycle or not do it at all. Sometimes you are bound by having to use async such as in my case where I had to consume a library that was implemented that way or use async but make the request synchronously as a compromise.

I'd recommend you spend some time reading about this. Here are few links to get your started if you confirm you are doing async in web forms.

https://msdn.microsoft.com/en-us/magazine/jj991977.aspx

https://blog.stephencleary.com/2012/02/async-and-await.html

https://blog.stephencleary.com/2012/07/dont-block-on-async-code.html

How and When to use `async` and `await`

An async/await example that causes a deadlock

Another thing you could do is also try to isolate when exactly IIS starts to die. This will help you pin point the code path or the page that trumps the error. Before you restart your IIS server again clear out your IIS logs. Next time it dies, look at the logs file carefully and see what pages were requested in the end. Additionally, what do you see in the event viewer? Sometimes a more details explanation is provided there.

BTW, what are doing displaying "yellow error page" in production? Please don't invite more trouble than you already have and use a custom error page and show a user friendly error message.

Good Luck and let us know what you find out.

Answer 3

It's difficult to say anything with certainty based on only your stack trace. But here is one possible explanation for the website being down.

IIS has something called Rapid fail protection, which it uses to try to protect itself from bad applications. When the Rapid fail protection is triggered, IIS shuts down the application pool running the bad application. When this happens, the application pool wont auto start even if users are trying to access the application. Only after the pool is manually started by an admin on the web server does the application come back on line.

One thing that can trigger the Rapid fail protection is repeated application crashes within a short time window. I think the default is 5 crashes in 5 minutes.

I think that your application is sometimes triggering the Rapid fail protection by throwing several unhandled exceptions in a short time window.

To fix the problem, you will have to code your application so that it will not throw unhandled exceptions. Adding an Application_Error handler to your Global.asax that handles and logs exceptions (that are not caught elsewhere) would be a good start.