CODEWITHSUNDEEP
c#async-awaitthreadpooldotnet-httpclient
Mark
Mark

Reputation: 5523

Unexplained timeouts and delays on getting the response from HttpClient SendAsync requests

We have a .NET 4.7.2 that is using a mixture of asynchronous and synchronous code (i understand its a no-no). We are using NancyFX on a windows service. The service gets rest calls and makes rest calls. Thread pool seems healthy (whole process only using 70 threads). For some reason, some http responses are being delayed by 10s of seconds and sometimes by 100s of seconds and leading to a task cancelation.

Here is how the code is structured

public async Task<Guid> SomeFunction()
{
   ...
   var response = await _httpClient.SendAsync(request, cancellationToken);
   ...
}

SomeFunction().Result

First, I thought for sure the response was being delayed on the network somewhere for some reason. But we've ruled that out multiple ways, most significantly by looking at the ETW trace through perfview and seeing the packet make it back almost instantly (using Microsoft-Windows-NDIS-PacketCapture/PacketFragment)

Secondly then, I was sure this had to do with threadpool issues caused by the .Result code on async methods. However, again, the threads on the process stay steady at 70 threads. Through perfview i can see that starvation is really not happening (using Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThreadAdjustment/Adjustment)

I've also thought maybe I'm hitting into a deadlock situation with await/async and .Result, but a deadlock would imply that the request would never complete, not that it would be delayed by 10s of seconds.

I've also double checked that we are only using one instance of httpclient and we indeed are.

What else could it be?

At this point we are removing the .Result and replacing it with proper async/await. But I have no proof that this will solve the issue since I'm not seeing any evidence of deadlocks or thread exhaustion.

Here is a perfview analysis enter image description here

We're also looking at the suggestion that we are exhausting http connections in some capacity. One reason i don't think its the case is that, according to perfview, the request is being sent out and packets make it back, but the response just doesn't make it up the c# stack. However these performance counters might indicate that there is some queuing going on.

enter image description here

Update We have increased the http connections with this and it seems to take effect.

<connectionManagement>
  <add address="*" maxconnection="1024"/>
</connectionManagement>

The queuing shown above is completely gone. However the problem of these requests not completing remains

Upvotes: 3

Views: 1808

Answers (2)

Mark
Mark

Reputation: 5523

Thanks for everyone who helped. In the end, nothing was a smoking gun. I couldn't prove thread starvation or http connection starvation. We ended up cleaning up our async/await code completed to remove things like .Result and added some caching and the problem went away.

My final guess is it's some edge case blocking in .net 4.7.2 when you mix and match async/await and .Result and have lots of requests going out.

Upvotes: 0

Stephen Cleary
Stephen Cleary

Reputation: 456887

What else could it be?

You've already checked my first guess, which would be thread pool starvation.

There's another possibility, depending on how the API calls are done. If there are many simultaneous requests to the same host, it is possible that the .NET networking stack may be throttling you. Non-ASP.NET apps have a default throttling of 2 simultaneous requests to the same host. In this case you have a server app but not an ASP.NET app, so you'd have that rather restrictive throttling on by default.

Recommendation: put this code in your startup:

ServicePointManager.DefaultConnectionLimit = int.MaxValue;

Note that .NET Core does not throttle client HTTP requests by default, so this is just emulating the behavior which is the default on modern .NET platforms.

Upvotes: 2

Related Questions