HttpClient.PostAsync Hanging Until All System.Timers Are Started

Question

I'm having an odd problem with HttpClient and Timers. I have a large number of objects (up to 10,000) that post to a web service. These objects are on timers and post to the service at some n time after creation. The problem is that the Post stalls, until all of the timers have started. See the code below for an example.

Q: Why does the Post hang until all Timers have started? How can it be fixed so that the post functions correctly while the other Timers start?

public class MyObject
{
    public void Run()
    {
        var result = Post(someData).Result; 
        DoOtherStuff();
    }
}

static async Task<string> Post(string data)
{
    using (var client = new HttpClient())
    {
        //Hangs here until all timers are started
        var response = await client.PostAsync(new Uri(url), data).ConfigureAwait(continueOnCapturedContext: false);
        var text = await response.Content.ReadAsStringAsync().ConfigureAwait(continueOnCapturedContext: false);

        return text;
    }
}

static void Main(string[] args)
{
    for (int i = 0; i < 1000; i++)
    {
        TimeSpan delay = TimeSpan.FromSeconds(1);
        if (i % 2 == 0) delay = TimeSpan.FromDays(1);

        System.Timers.Timer timer = new System.Timers.Timer();
        timer.AutoReset = false;
        timer.Interval = delay.TotalMilliseconds;
        timer.Elapsed += (x, y) =>
            {
                MyObject o = new MyObject();
                o.Run();
            };

        timer.Start();
    }

    Console.ReadKey();
}

Answer 1

Because you're using up all the ThreadPool threads.

There's a lot wrong with your sample code. You're killing any chance of having reasonable performance, not to mention that the whole thing is inherently unstable.

You're creating a thousand timers in a loop. You're not keeping a reference to any of them, so they will be collected the next time the GC runs - so I'd expect that in practice, very few of them will actually run, unless there's very little memory allocated until they actually get to run.

The timer's Elapsed event will be invoked on a ThreadPool thread. In that thread, you synchronously wait for a bunch of asynchronous calls to complete. That means you're now wasting a thread pool thread, and completely wasting the underlying asynchronicity of the asynchronous method.

The continuation to the asynchronous I/O will be posted to ThreadPool as well - however, the ThreadPool is full of timer callbacks. It will slowly start creating more and more threads to accomodate the amount of work scheduled, until it finally is able to execute the first callback from the asynchronous I/O and it slowly untangles itself. At this point, you likely have more than 1000 threads, and are showing a complete misunderstanding of how to do asynchronous programming.

One way (still rather bad) to fix both problems is to simply make use of asynchronicity all the time:

public class MyObject
{
    public async Task Run()
    {
        var result = await Post(someData);
        DoOtherStuff();
    }
}

static async Task<string> Post(string data)
{
    using (var client = new HttpClient())
    {
        //Hangs here until all timers are started
        var response = await client.PostAsync(new Uri(url), new StringContent(data)).ConfigureAwait(continueOnCapturedContext: false);
        var text = await response.Content.ReadAsStringAsync().ConfigureAwait(continueOnCapturedContext: false);

        return text;
    }
}

static void Main(string[] args)
{
    var tasks = new List<Task>();

    for (int i = 0; i < 1000; i++)
    {
        TimeSpan delay = TimeSpan.FromSeconds(1);
        if (i % 2 == 0) delay = TimeSpan.FromDays(1);

        tasks.Add(Task.Delay(delay).ContinueWith((_) => new MyObject().Run()));
    }

    Task.WaitAll(tasks.ToArray());

    Console.WriteLine("Work done");
    Console.ReadKey();
}

A much better way would be to implement some scheduler that handles dispatching the asynchronous I/O with the throttling you need. You probably want to limit the number of concurrent requests or something like that, rather than running the requests in pre-defined intervals (and ignoring the fact that some requests might take very long, timeout or some such).

Answer 2

As mentioned in another reply, that Result property is the problem. when you are using it, asyc will be come sync. If you want to run async operations in console or windows service applications, try Nito AsyncEx library. It creates an AsyncContext. Now you can change void Run to Task Run which is await-able and doesn't need the blocking Result property and in this case await Post will work in the Run method.

static void Main(string[] args)
{
    AsyncContext.Run(async () =>
    {
            var data = await ...;
    });
}

Answer 3

It's because the timers are set up very quickly so they have all finished setting up by the time PostAsync completes. Try putting a Thread.Sleep(1000) after timer.Start, this will slow down the set up of your timers and you should see some PostAsync executions complete.

By the way, Task.Result is a blocking operation, which can cause a deadlock when run from a GUI application. There is more information in this article.

Answer 4

As you're running on a console application, which uses the default ThreadPoolSynchronizationContext, you shouldn't really be experiencing the "hanging" feeling as if you are in a UI application. I assume its because Post is taking longer to return than to allocate 1000 timers.

In order for your method to run async, it has to go "async all the way. Using the Task.Result property, as mentioned before will simply block the asynchronous operation until it completes.

Lets see what we need to do for this to be "async all the way":

First, lets change Run from void to async Task so we can await on the Post method:

public async Task Run()
{
    var result = await Post(someData);
    DoOtherStuff();
}

Now, since Run became awaitable, as it returns a Task, we can turn Timer.Elapsed to an async event handler and await on Run.

static void Main(string[] args)
{
   for (int i = 0; i < 1000; i++)
   {
       TimeSpan delay = TimeSpan.FromSeconds(1);
       if (i % 2 == 0) delay = TimeSpan.FromDays(1);

       System.Timers.Timer timer = new System.Timers.Timer();
       timer.AutoReset = false;
       timer.Interval = delay.TotalMilliseconds;
       timer.Elapsed += async (x, y) =>
       {
            MyObject o = new MyObject();
            await o.Run();
       };

        timer.Start();
   }

   Console.ReadKey();
}

That's it, now we flow async all the way down to the HTTP request and back.