CODEWITHSUNDEEP
c#dotnet-httpclient
Paul Rozhkin
Paul Rozhkin

Reputation: 81

C# HttpClient.GetAsync ignore HttpCompletionOption.ResponseHeadersRead

I got a weird situation when using C# HttpClient. I am trying to use the HttpCompletionOption.ResponseHeadersRead option in GetAsync to get response headers without content as quickly as possible. But when downloading files, I am in await GetAsync until the whole content is downloaded over the network (i checked this with Fiddler). I am attaching an example code that downloads a 1Gb test file. The example application will hang in the await client.GetAsync until all file content is received over the network. How do I get control back when the headers have finished receiving and not wait for the complete content transfer over the network?

using System;
using System.IO;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

public class Program
    {
        private const int HttpBufferSize = 81920;

        private static async Task Main(string[] args)
        {
            var url = new Uri("http://212.183.159.230/1GB.zip");

            await DownloadFileAsync(@"C:\1GB.zip", url, CancellationToken.None).ConfigureAwait(false);
        }

        private static async Task DownloadFileAsync(string filePath, Uri fileEndpoint,
            CancellationToken token)
        {
            using var client = new HttpClient();

            using var response = await client.GetAsync(fileEndpoint, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false);

            response.EnsureSuccessStatusCode();

            await using var contentStream = await response.Content.ReadAsStreamAsync(token).ConfigureAwait(false);
            await using var stream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None);
            await contentStream.CopyToAsync(stream, HttpBufferSize, token).ConfigureAwait(false);
        }
    }

Upvotes: 2

Views: 3781

Answers (2)

Paul Rozhkin
Paul Rozhkin

Reputation: 81

I have identified the reason for this behavior. The reason was Fiddler. It acted as a proxy and did not seem to redirect partially received responses. To check this, I've added console output for each of the operations:

Console.WriteLine($"Start GetAsync - {DateTime.Now}");
using var response = await client.GetAsync(fileEndpoint, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false);
Console.WriteLine($"End GetAsync - {DateTime.Now}");

response.EnsureSuccessStatusCode();

await using var contentStream = await response.Content.ReadAsStreamAsync(token).ConfigureAwait(false);
await using var stream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None);

Console.WriteLine($"Start CopyToAsync - {DateTime.Now}");
await contentStream.CopyToAsync(stream, HttpBufferSize, token).ConfigureAwait(false);
Console.WriteLine($"End CopyToAsync - {DateTime.Now}");

Results with running program Fiddler:

Start GetAsync - 30.06.2021 17:46:03
End GetAsync - 30.06.2021 17:46:49
Start CopyToAsync - 30.06.2021 17:46:49
End CopyToAsync - 30.06.2021 17:46:51

Results without Fiddler:

Start GetAsync - 30.06.2021 17:38:32
End GetAsync - 30.06.2021 17:38:32
Start CopyToAsync - 30.06.2021 17:38:32
End CopyToAsync - 30.06.2021 17:39:48

Conclusion: be careful with proxies

Upvotes: 3

Riddell
Riddell

Reputation: 1489

You are sending a GET request. If you only require the headers then you can use HEAD request. An example for HttpClient:

client.SendAsync(new HttpRequestMessage(HttpMethod.Head, url))

Caution: Servers can block HEAD requests so make sure to handle gracefully. For example, fallback to GET request if the response fails but it will be at the cost of speed.

Upvotes: 3

Related Questions