CODEWITHSUNDEEP
c#multithreadingasynchronous
ebb
ebb

Reputation: 9377

Number of web requests limited

Using the following code:

static void Main(string[] args)
{
    ServicePointManager.DefaultConnectionLimit = 1000;

    var client = new HttpClient();
    var uris = File.ReadAllLines(@"C:\urls.txt").Select(x => new Uri(x));

    foreach(var uri in uris)
    {
        var url = uri.ToString();

        var task = client.GetStringAsync(uri);
        task.ContinueWith(t => Console.WriteLine("Done {0}", url), TaskContinuationOptions.OnlyOnRanToCompletion);
        task.ContinueWith(t => Console.WriteLine("Failed {0}", url), TaskContinuationOptions.OnlyOnFaulted);
        task.ContinueWith(t => Console.WriteLine("Cancelled {0}", url), TaskContinuationOptions.OnlyOnCanceled);
    }

    Console.ReadKey();
}

I can at best request 15-20 urls concurrently, according to fiddler. All of these urls are unique and not pointing to the same host.

What is going on?

Upvotes: 2

Views: 155

Answers (1)

pwnyexpress
pwnyexpress

Reputation: 1016

How many cores does the CPU on the machine your running this on have? There are limits to how many concurrent operations your machine can handle. Also, TPL automatically decides the right amount of parallelism to invoke given the task at hand. It is not always more efficient to spin up 1000 threads to accomplish a task. There's substantial overheard managing messages being passed between threads.

This may not have any performance improvements, but this should be more idiomatic for parallelism:

static void Main(string[] args)
{
    ServicePointManager.DefaultConnectionLimit = 1000;

    var uris = File.ReadAllLines(@"C:\urls.txt").Select(x => new Uri(x));

    foreach(var uri in uris)
    {
        var client = new HttpClient();
        var url = uri.ToString();

        var task = client.GetStringAsync(uri);
        task.ContinueWith(t => Console.WriteLine("Done {0}", url), TaskContinuationOptions.OnlyOnRanToCompletion);
        task.ContinueWith(t => Console.WriteLine("Failed {0}", url), TaskContinuationOptions.OnlyOnFaulted);
        task.ContinueWith(t => Console.WriteLine("Cancelled {0}", url), TaskContinuationOptions.OnlyOnCanceled);
    }

    Console.ReadKey();
}

or even:

static void Main(string[] args)
{
    ServicePointManager.DefaultConnectionLimit = 1000;

    var uris = File.ReadAllLines(@"C:\urls.txt").Select(x => new Uri(x));

    Parallel.ForEach(uris, uri => {
        WebRequest myRequest = WebRequest.Create(uri.toString());
        // handle response synchronously
    });

}

Upvotes: 1

Related Questions