CODEWITHSUNDEEP
c#async-awaittimeouthttpclient
LukAss741
LukAss741

Reputation: 831

HttpClient query occasionally hangs

I initialise HttpClient like so:

public static CookieContainer cookieContainer = new CookieContainer();
public static HttpClient httpClient = new HttpClient(new HttpClientHandler() { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate, CookieContainer = cookieContainer }) { Timeout = TimeSpan.FromSeconds(120) };

so all queries should throw TaskCanceledException if no response is received within 120 seconds. But some queries (like 1 of 100 000-1 000 000) hang infinitely.

I wrote following code:

public static async Task<HttpResponse> DownloadAsync2(HttpRequestMessage httpRequestMessage)
{
    HttpResponse response = new HttpResponse { Success = false, StatusCode = (int)HttpStatusCode.RequestTimeout, Response = "Timeout????????" };
    Task task;
    if (await Task.WhenAny(
        task = Task.Run(async () =>
        {
            try
            {
                HttpResponseMessage r = await Global.httpClient.SendAsync(httpRequestMessage).ConfigureAwait(false);
                response = new HttpResponse { Success = true, StatusCode = (int)r.StatusCode, Response = await r.Content.ReadAsStringAsync().ConfigureAwait(false) };
            }
            catch (TaskCanceledException)
            {
                response = new HttpResponse { Success = false, StatusCode = (int)HttpStatusCode.RequestTimeout, Response = "Timeout" };
            }
            catch (Exception ex)
            {
                response = new HttpResponse { Success = false, StatusCode = -1, Response = ex.Message + ": " + ex.InnerException };
            }
        }),
        Task.Run(async () =>
        {
            await Task.Delay(TimeSpan.FromSeconds(150)).ConfigureAwait(false);
        })
    ).ConfigureAwait(false) != task)
    {
        Log("150 seconds passed");
    }
    return response;
}

which actually occasionally executes Log("150 seconds passed");.

I call it like so:

HttpResponse r = await DownloadAsync2(new HttpRequestMessage
{
    RequestUri = new Uri("https://address.com"),
    Method = HttpMethod.Get
}).ConfigureAwait(false);

Why TaskCanceledException sometimes isn't thrown after 120 seconds?

Upvotes: 5

Views: 2884

Answers (6)

Misha
Misha

Reputation: 1826

OK, I officially give up! I'm replacing the code with:

    try
    {
        return Task.Run(() => httpClient.SendAsync(requestMessage)).Result;
    }
    catch (AggregateException e)
    {
        if (e.InnerException != null)
            throw e.InnerException;
        throw;
    }

Upvotes: 0

Misha
Misha

Reputation: 1826

And the answer is:

ThreadPool.SetMinThreads(MAX_THREAD_COUNT, MAX_THREAD_COUNT);

where MAX_THREAD_COUNT is some number (I use 200). You MUST set at least the second parameter (completionPortThreads), and most probably the first (workerThreads). I had already set the first, but not the second, and now that it is working I am keeping both set.

Alas, this isn't the answer. See comments below

Upvotes: 0

L.Vallet
L.Vallet

Reputation: 1052

With Flurl, you can configure Timeout per client, per request or globally.


// call once at application startup
FlurlHttp.Configure(settings => settings.Timeout = TimeSpan.FromSeconds(120));

string url = "https://address.com";

// high level scenario
var response = await url.GetAsync();

// low level scenario
await url.SendAsync(
    HttpMethod.Get, // Example
    httpContent, // optional
    cancellationToken,  // optional
    HttpCompletionOption.ResponseHeaderRead);  // optional

// Timeout at request level
await url
    .WithTimeout(TimeSpan.FromSeconds(120))
    .GetAsync();

Fluent HTTP documentation

Flurl configuration documentation

Upvotes: 1

Alon Catz
Alon Catz

Reputation: 2530

On Windows & .NET, the number of concurrent outgoing HTTP request to the same endpoint is limited to 2 (as per HTTP 1.1 specification). If you create a ton of concurrent requests to the same endpoint they will queue up. That is one possible explanation to what you experience.

Another possible explanation is this: you don't set the Timeout property of HttpClient explicitly, so it defaults to 100 seconds. If you keep making new requests, while the previous ones didn't finish, system resources will become used up.

I suggest setting the Timeout property to a low value - something proportional to the frequency of the calls you make (1 sec?) and optionally increasing the number of conncurrent outgoing connections with ServicePointManager.DefaultConnectionLimit

Upvotes: 3

LukAss741
LukAss741

Reputation: 831

I discovered it was httpClient.SendAsync method which occasionally hangs. Therefore I added a cancellation token set to X seconds. But even with a cancelletion token it may sometimes remain stuck and never throw TaskCanceledException.

Therefore I proceeded to workaround that keeps the SendAsync task forever stuck on background and continue with other work.

Here is my workaround:

public static async Task<Response> DownloadAsync3(HttpRequestMessage httpRequestMessage, string caller)
{
    Response response;
    try
    {
        using CancellationTokenSource timeoutCTS = new CancellationTokenSource(httpTimeoutSec * 1000);
        using HttpResponseMessage r = await Global.httpClient.SendAsync(httpRequestMessage, HttpCompletionOption.ResponseContentRead, timeoutCTS.Token).WithCancellation(timeoutCTS.Token).ConfigureAwait(false);
        response = new Response { Success = true, StatusCode = (int)r.StatusCode, Message = await r.Content.ReadAsStringAsync().ConfigureAwait(false) };
    }
    catch (TaskCanceledException)
    {
        response = new Response { Success = false, StatusCode = (int)HttpStatusCode.RequestTimeout, Message = "Timeout" };
    }
    catch (Exception ex)
    {
        response = new Response { Success = false, StatusCode = -1, Message = ex.Message + ": " + ex.InnerException };
    }
    httpRequestMessage.Dispose();
    return response;
}

public static Task<T> WithCancellation<T>(this Task<T> task, CancellationToken cancellationToken)
{
    return task.IsCompleted
        ? task
        : task.ContinueWith(
            completedTask => completedTask.GetAwaiter().GetResult(),
            cancellationToken,
            TaskContinuationOptions.ExecuteSynchronously,
            TaskScheduler.Default);
}

Upvotes: 1

Arman Ebrahimpour
Arman Ebrahimpour

Reputation: 4461

I don't know with how frequency you call DownloadAsync2, but your code smells a lot for bursting and starving ThreadPool.

Initial number of threads in ThreadPool by default is limited to number of CPU logical cores (usually 12 for today normal systems) and in case of unavailability of threads in ThreadPool, 500ms takes for each new thread to be generated.

So for example:

for (int i = 0; i < 1000; i++)
{
    HttpResponse r = await DownloadAsync2(new HttpRequestMessage
    {
        RequestUri = new Uri("https://address.com"),
        Method = HttpMethod.Get
    }).ConfigureAwait(false);
}

This code with a high chance will be freezed, specially if you have some lock or any cpu intensive tasks somewhere in your code. Because you invoke new thread per calling DownloadAsync2 so all threads of ThreadPool consumed and many more of them still needed.

I know maybe you say "all of my tasks have been awaited and they release for other works". but they also consumed for starting new DownloadAsync2 threads and you will reach the point that after finishing await Global.httpClient.SendAsync no thread remains for re-assigning and completing the task.

So method have to wait until one thread being available or generated to complete (even after timeout). Rare but feasible.

Upvotes: 3

Related Questions