CODEWITHSUNDEEP
c#multithreadinghttpthreadpool
David Hawkins
David Hawkins

Reputation: 1067

HttpClient.PostAsync There were not enough free threads in the ThreadPool to complete the operation

I call the below code once every second to poll a camera but after running for a day or two, it stops working. 

public List<CameraEvent> GetEventsSince(CaptureTime afterDateTime)
{
    string uri = string.Format(
                    "http://{0}/ISAPI/channels/1/events/detect", _ipAddress);
    using (var client = new HttpClient())
    {
        client.Timeout = TimeSpan.FromSeconds(5);
        AddBasicAuth(client);

        try
        {
            HttpResponseMessage response =
                client.PostAsync(
                    uri, new StringContent(GetPicTimeXml(afterDateTime))).Result;

            logger.Debug(
                string.Format("Status code response={0}", response.StatusCode));

            if (response.StatusCode == HttpStatusCode.Unauthorized ||
                response.StatusCode == HttpStatusCode.Forbidden)
            {
                // 401
                currentState = 2;
                return new List<CameraEvent>();
            }

            if (response.StatusCode == HttpStatusCode.OK)
            {
                // OK
                currentState = 0;
            }
            List<CameraEvent> events = new CameraHttpResponseHandler()
                                           .HandleHttpResponse(response);
            AppendPlateImages(events);
            return events;
        }
        catch (AggregateException ex)
        {
            //if (ex.InnerException is TaskCanceledException)
            //{
            //    // Timeout
            //    currentState = 1;
            //}
            logger.Error("AggregateException", ex);
        }
        catch (Exception ex)
        {
            logger.Error("Generic exception", ex);
        }

        return new List<CameraEvent>();
    }
}

The error I get is:

2015-08-17 07:59:57,310 [16] ERROR CameraHttpClient AggregateException System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: There were not enough free threads in the ThreadPool to complete the operation.

The parent thread calling GetEventsSince is a background worker thread running in a loop if that makes any difference.

Has anyone seen this issue or have any suggestions on what might be causing threads to be used up?

Upvotes: 1

Views: 4077

Answers (1)

Todd Menier
Todd Menier

Reputation: 39369

Hard to say for certain, but if the root cause of the threadpool starvation is this method, then it's a great example of why asynchronous code is beneficial on the server.

HttpClient is an asynchronous API, meaning that if you properly await calls, you free up a thread and send it back to the threadpool until the call returns. By calling .Result, you are blocking the thread for the entire duration of the call. Say this method takes several seconds beginning to end, and that 99.9% of that time is waiting on I/O (not an unreasonable guess). The way it is, you are consuming a thread for 100% of that time. If you refactor it to run asynchronously, your thread consumption drops to 0.1% of the time, and the threadpool is suddenly much fuller on average.

So I would begin by marking the method async (use Task<List<CameraEvent>> as the return type) and using await instead of .Result where asynchronous APIs are used. I don't know what CameraHttpResponseHandler.HandleHttpResponse does exactly, but I'm guessing there's blocking on I/O there too and it should be converted as well as well and called using await.

This has implications on how the root application calls this method. I'd need to see that code to suggest the best approach. TPL Dataflow might be a good fit here - not only is it helpful in calling async methods at regular intervals, but it also supports throttling concurrency as sort of a safeguard against problems like this.

Upvotes: 3

Related Questions