CODEWITHSUNDEEP
c#
zackmark15
zackmark15

Reputation: 727

HttpClient high data usage when getting remote file size

I'm getting the file size of remote urls and I just noticed the difference between HttpClient and httpWebRequest.

I compared and I noticed that httpclient is taking too much data. this is a big issue for me because, in the Philippines we are only have limited data

Could you please tell me what's wrong with my httpclient class? I can't figure out what is causing the high data usage

HttpClient

       HttpClientHandler handler = new HttpClientHandler()
        {
            Proxy = null,
            UseProxy = false,
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
        };

        var client = new HttpClient(handler);

        client.DefaultRequestHeaders.Add("Method", "GET");
        client.DefaultRequestHeaders.Add("Referer", uriPath.AbsoluteUri);
        client.DefaultRequestHeaders.Add("Origin", "https://www.samplesite.com");
        client.DefaultRequestHeaders.ConnectionClose = true;
        client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36");
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));

        using (HttpResponseMessage response = await client.GetAsync(uriPath, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false))
        {
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadAsStringAsync();
            var resultTask = response.Content.ReadAsStringAsync();
            var timeoutTask = Task.Delay(3000);

            var completed = await Task.WhenAny(resultTask, timeoutTask).ConfigureAwait(false);

            if (completed == timeoutTask)
                return null;

            return await resultTask;
        }

HttpWebRequest

      var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
        webRequest.Method = "HEAD";
        webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36";

        using (var webResponse = await webRequest.GetResponseAsync())
        {
            return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
        }

Upvotes: 0

Views: 773

Answers (2)

zackmark15
zackmark15

Reputation: 727

Finally it's solved Big thanks to @Dmitry

Here's the HttpClient updated code

   public static async Task<string> GetTotalBytes(Uri uriPath)
    {
        HttpClientHandler handler = new HttpClientHandler();
        handler.Proxy = null;

        using (var client = new HttpClient(handler))
        {
            HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, uriPath);
            using (var response = await client.SendAsync(message))
            {
                response.EnsureSuccessStatusCode();
                var lenght = response.Content.Headers.ContentLength;
                return lenght.ToString();
            }
        }
     }

RESULT (HttpClient):

Total URL: 355
Filsize: 1.14 GB
Elapsed Time: 0h 0m 2s 51.3ms

and here's HttpWebRequest (Maybe someone will need)

        var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
        webRequest.Method = "HEAD";
        webRequest.Proxy = null;

        using (var webResponse = await webRequest.GetResponseAsync())
        {
            return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
        }

Upvotes: 0

Dmitry Kolchev
Dmitry Kolchev

Reputation: 2216

You are using different HTTP methods GET in case of HttpClient & HEAD in case of WebRequest. To get file size you will enough HEAD method in both cases

The HTTP GET method requests a representation of the specified resource. Requests using GET should only retrieve data.

The HTTP HEAD method requests the headers that are returned if the specified resource would be requested with an HTTP GET method. Such a request can be done before deciding to download a large resource to save bandwidth, for example.

You need to change this code line

 client.DefaultRequestHeaders.Add("Method", "GET");

it MUST BE

 client.DefaultRequestHeaders.Add("Method", "HEAD");

A response to a HEAD method does not have a body in contradistinction to GET

UPD: use SendAsync method (not GetAsync)

HttpClientHandler handler = new HttpClientHandler();
using var client = new HttpClient(handler);

string requestUri = "enter request uri";

HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, requestUri);
using var response = await client.SendAsync(message);

Upvotes: 3

Related Questions