async HttpWebRequest vs sync in multithreaded app

Question

I have an app, where i need to make alot httpWebRequest at one time and get response as fast as it possible for each of them. Actualy i don't need to get all responses fast, i need to get each response as fast as i can. I mean that there must be minimum time between i send each request ang get response.

I read many articles about async/await technology and chose this solution:

    private static void Main()
    {
        ServicePointManager.DefaultConnectionLimit = 100;
        ServicePointManager.Expect100Continue = false;
        ServicePointManager.UseNagleAlgorithm = false;

        ProcessAsync("https://www.google.ru");

        Console.ReadKey();
    }

    private static void ProcessAsync(string url)
    {
        for (var i = 1; i <= 100; i++)
            Task.Factory.StartNew(async () => await ProcessInternalAsync(url));
    }

    private static async Task ProcessInternalAsync(string url)
    {
        var request = GetRequest(url);

        var sw = Stopwatch.StartNew();
        await GetStringContentAsync(request);
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);
    }

    public static async Task<string> GetStringContentAsync(HttpWebRequest webRequest)
    {
        using (var response = (HttpWebResponse) await webRequest.GetResponseAsync()
                                                                .ConfigureAwait(false))
        {
            var content = await GetStringContentFromResponseAsync(response)
                .ConfigureAwait(false);
            return content;
        }
    }

    private static async Task<string> GetStringContentFromResponseAsync(HttpWebResponse response)
    {
        using (var responseStream = GetStreamForResponse(response))
        {
            if (responseStream == null)
                return null;
            using (var streamReader = new StreamReader(responseStream))
            {
                var content = await streamReader.ReadToEndAsync()
                                                .ConfigureAwait(false);
                return content;
            }
        }
    }    

And this works, but wery slow.. (500-1600 ms each request)

So i tried "simple" solution without async/await:

    private static void Main()
    {
        ServicePointManager.DefaultConnectionLimit = 100;
        ServicePointManager.Expect100Continue = false;
        ServicePointManager.UseNagleAlgorithm = false;

        Process("https://www.google.ru");

        Console.ReadKey();
    }

    private static void Process(string url)
    {
        for (var i = 1; i <= 100; i++)
            Task.Factory.StartNew(() => ProcessInternal(url));
    }

    private static void ProcessInternal(string url)
    {
        var request = GetRequest(url);

        var sw = Stopwatch.StartNew();
        GetStringContent(request);
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);
    }

    public static string GetStringContent(HttpWebRequest webRequest)
    {
        using (var response = (HttpWebResponse) webRequest.GetResponse())
        {
            var content = GetStringContentFromResponse(response);
            return content;
        }
    }

    private static string GetStringContentFromResponse(HttpWebResponse response)
    {
        using (var responseStream = GetStreamForResponse(response))
        {
            if (responseStream == null)
                return null;
            using (var streamReader = new StreamReader(responseStream))
            {
                var content = streamReader.ReadToEnd();
                return content;
            }
        }
    }    

And it works much faster! (55-180 ms each request).

Here are some other methods:

    private static HttpWebRequest GetRequest(string url)
    {
        var request = (HttpWebRequest) WebRequest.Create(url);
        request.Timeout = 15000;
        request.Proxy = null;
        request.Headers.Add("Accept-Encoding", "gzip,deflate");

        return request;
    }

    private static Stream GetStreamForResponse(HttpWebResponse webResponse)
    {
        var responseStream = webResponse.GetResponseStream();
        if (responseStream == null)
            return null;

        Stream stream;
        switch (webResponse.ContentEncoding.ToUpperInvariant())
        {
            case "GZIP":
                stream = new GZipStream(responseStream, CompressionMode.Decompress);
                break;
            case "DEFLATE":
                stream = new DeflateStream(responseStream, CompressionMode.Decompress);
                break;
            default:
                stream = responseStream;
                break;
        }
        return stream;
    }

So my question is: why is so big difference? I thought that when I'm using await with I/O operations - this is a good way and it is the best way to do HTTP requests. But in practice it is not fully true.

In solution where there are not async/await we have thread block when waiting response and if it will not be Google and will be a server with long running operations this requests will block all my app.

May be there is better way to solve my problem?

P.S.: This "magic": ServicePointManager.Expect100Continue = false; ServicePointManager.UseNagleAlgorithm = false;
is just some advices from internet and i have no idea how it works or not..

Update Thanks to Servy, i finaly understood my fault in measuaring. Now Process method looks like:

    private static void Process(string url)
    {
        var taskCollection = new List<Task>();
        for (var i = 1; i <= 100; i++)
        {
            var sw = Stopwatch.StartNew();
            taskCollection.Add(Task.Factory.StartNew(() =>
                                                     {
                                                         ProcessInternal(url);
                                                         sw.Stop();
                                                         Console.WriteLine(sw.ElapsedMilliseconds);
                                                     }));
        }
        Task.WaitAll(taskCollection.ToArray());
    }

Now second solution shows its milliseconds (300-1000ms for request). But anyway it is faster, than async/await. Does async/await method must be slower?

Answer 1

Use the stopwatch to measure the time of the entire operation, and you'll find that the times will be much closer to each other.

Basically, you're starting your stopwatch in the asynchronous version sooner, which is why they're appearing to take so much longer.

In the asynchronously solution you start every single operation synchronously, and then asynchronously wait for them all to finish. In this case the stopwatches are all started pretty much right away when the application starts.

For your synchronous solution you're not even starting the stopwatch until each of the task gets scheduled to run in a thread pool thread, gets allocated to a thread, and is actually run from that thread. Much of the waiting is going to be happening here and you haven't even started your stopwatch yet.