CODEWITHSUNDEEP
.netmultithreadingdotnet-httpclient
Mickael Magniez
Mickael Magniez

Reputation: 111

.NET HttpClient multithreading

I have some trouble using HttpClient in multiple thread.

If i launch many download simultaneously, the first download of each thread is very slow (and increase with parallel threads)

For example, if I have one thread, everything is fine

First download Elapsed Time Download: 197 ms
Second download Elapsed Time Download: 171 ms

But with one thread, download time increase

First download Elapsed Time Download: 3881 ms
...
Second download Elapsed Time Download: 96 ms
...

Network bandwith is not an issue, i have same problem with localhost.

Here is some code to reproduce problem :

static void Main(string[] args)
{
    ServicePointManager.DefaultConnectionLimit = 200;
    List<Task> tasks = new List<Task>();
    for (var i = 0; i < 10; i++)
    {
        tasks.Add(
            Task.Factory.StartNew(() =>
            {
                Stopwatch s = Stopwatch.StartNew();
                HttpClient httpClient = new HttpClient();
                HttpResponseMessage httpDownloadResponse = httpDownloadResponse = httpClient.GetAsync("http://www.google.fr/", HttpCompletionOption.ResponseHeadersRead).Result;
                s.Stop();
                Console.WriteLine("First download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds);

                s = Stopwatch.StartNew();
                httpClient = new HttpClient();
                httpDownloadResponse = httpClient.GetAsync("http://www.google.fr/", HttpCompletionOption.ResponseHeadersRead).Result;
                s.Stop();
                Console.WriteLine("Second download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds);
            })
        );
    }
    Task.WaitAll(tasks.ToArray());
    while (Console.ReadLine() != null) ;
}

Upvotes: 2

Views: 4154

Answers (1)

Brian Reischl
Brian Reischl

Reputation: 7356

I can't actually reproduce your problem on my machine. No matter what I do, the first request always takes about the same time, and second requests are faster.

My best guess is that you're exhausting the threadpool. Each time you call Task.Factory.StartNew it starts a new Task, which takes a new threadpool thread. Then you call HttpClient.GetAsync, which starts another Task on another thread and blocks the first thread. So you're taking up two threads for each request. If you do that enough you'll use all the threads in the threadpool, and requests will start queuing up. The threadpool will add more threads, but slowly - typically one new thread every 0.5 seconds. So that might be part of it.

Also, you're using HttpClient wrong. Each HttpClient instance holds a connection pool, so you generally want to create one instance and reuse it. Here is some cleaner code - try it and see if it solves your problem.

public static void Main()
{
    ServicePointManager.DefaultConnectionLimit = 200;
    List<Task> tasks = new List<Task>();
    using (var client = new HttpClient())
    {
        for (var i = 0; i < 10; i++)
        {
            tasks.Add(DoRequest(i, client));
        }

        Task.WaitAll(tasks.ToArray());
    }
}

private async Task DoRequest(int id, HttpClient client)
{
    const string url = "http://www.google.fr/";
    Stopwatch s = Stopwatch.StartNew();
    HttpResponseMessage httpDownloadResponse = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
    s.Stop();
    Console.WriteLine("Task {1} - First download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds, id);

    s = Stopwatch.StartNew();
    httpDownloadResponse = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
    s.Stop();
    Console.WriteLine("Task {1} - Second download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds, id);

}

Upvotes: 2

Related Questions