CODEWITHSUNDEEP
c#asp.net-corehttphttpclient.net-9.0
Joshua Rogers
Joshua Rogers

Reputation: 3548

HttpClient hanging while fetching Url

I'm trying to ping this url for live events https://www.imf.org/en/live with this code:

using var httpClientHandler = new HttpClientHandler() { MaxAutomaticRedirections = 10, AllowAutoRedirect = true};
using var httpClient = new HttpClient(httpClientHandler);
var httpRequest = new HttpRequestMessage(HttpMethod.Get, "https://www.imf.org/en/live");    

httpRequest.Headers.TryAddWithoutValidation("authority", "www.imf.org");
httpRequest.Headers.TryAddWithoutValidation("method", "GET");    
httpRequest.Headers.TryAddWithoutValidation("scheme", "https");
httpRequest.Headers.TryAddWithoutValidation("accept", "text/html");
httpRequest.Headers.TryAddWithoutValidation("accept-encoding", "gzip");
httpRequest.Headers.TryAddWithoutValidation("accept-language", "en-US,en;q=0.9");
httpRequest.Headers.TryAddWithoutValidation("cache-control", "no-cache");
httpRequest.Headers.TryAddWithoutValidation("pragma", "no-cache");

var response = await httpClient.SendAsync(httpRequest);
response.EnsureSuccessStatusCode();

In a .net9 Console application but it gets stuck or yields and error

{"Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.."}

If I try the same httprequest from powershell, it works fine and returns some content

 Invoke-WebRequest -UseBasicParsing -Uri "https://www.imf.org/en/live" `
-Headers @{  
  "authority"="www.imf.org"
  "method"="GET" 
  "scheme"="https"
  "accept"="text/html"
  "accept-encoding"="gzip"
  "accept-language"="en-US,en;q=0.9"
  "cache-control"="no-cache"
  "pragma"="no-cache"    
}


StatusCode        : 200
StatusDescription : OK
Content           :


                    <!DOCTYPE html>
                    <html lang="en" xmlns="http://www.w3.org/1999/xhtml">
                    <head>
                        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />

Any ideas why the c# equivalent doesn't work? I tried to play with the HttpClientHandler and the SocketsHttpHandler for hours trying to tweak configuration variables with no success. Thanks!

Upvotes: 1

Views: 54

Answers (1)

Guru Stron
Guru Stron

Reputation: 142943

It seems that the site needs some extra headers (I assume that Powershell's Invoke-WebRequest adds them automatically):

httpRequest.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36");
httpRequest.Headers.TryAddWithoutValidation("sec-ch-ua-mobile", "0");
httpRequest.Headers.TryAddWithoutValidation("sec-ch-ua-platform", "Windows");
httpRequest.Headers.TryAddWithoutValidation("sec-fetch-dest", "document");

Also there is no need for headers like method since they should be added from the request (you specify request method there). The full set of headers I've used:

httpRequest.Headers.TryAddWithoutValidation("accept", "text/html");
httpRequest.Headers.TryAddWithoutValidation("accept-encoding", "gzip");
httpRequest.Headers.TryAddWithoutValidation("accept-language", "en-US,en;q=0.9");
httpRequest.Headers.TryAddWithoutValidation("cache-control", "no-cache");
httpRequest.Headers.TryAddWithoutValidation("pragma", "no-cache");

// added:
httpRequest.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36");
httpRequest.Headers.TryAddWithoutValidation("sec-ch-ua-mobile", "0");
httpRequest.Headers.TryAddWithoutValidation("sec-ch-ua-platform", "Windows");
httpRequest.Headers.TryAddWithoutValidation("sec-fetch-dest", "document");

Also I recommend to add the automatic decompression to the handler (since accept-encoding is set to gzip):

var httpClientHandler = new HttpClientHandler() 
{ 
    // ... 
    , AutomaticDecompression = DecompressionMethods.All // or GZip
};

And also check out the Use IHttpClientFactory to implement resilient HTTP requests article - depending on the use-case the disposal of the HttpClient may be highly discouraged.

Upvotes: 1

Related Questions