CODEWITHSUNDEEP
c#.netsslhttpwebrequest
nir weiner
nir weiner

Reputation: 305

Https GET request passes in Postman and browser but fails in code

I'm trying to obtain a JSON via a rest API using, targeting .Net 4.5
I've tried various methods in code, but the all end up in me getting:

"Authentication failed because the remote party has closed the transportstream" .

the exact same URL works via browser and Postman.

So far, I've tried using .Net's WebClient, HttpClient and HttpWebRequest with identical results. I've tried comparing requests between Postman and my code (via RequestBin), but even when they were identical, I still kept getting back:

Authentication failed because the remote party has closed the transport

My current code is using HttpWebRequest, but every solution will do.

I've played around with all of the security protocols, some of them will cause the API to return 404 and some will cause the server to return

"Authentication failed because the remote party has closed the transport stream".

Here's my current code:

public string GetCityStreets()
{
    var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
    var request = (HttpWebRequest)WebRequest.Create(url);
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 |
                                           SecurityProtocolType.Tls12 |
                                           SecurityProtocolType.Tls11 |
                                           SecurityProtocolType.Tls;

    var response = (HttpWebResponse)request.GetResponse();
    string jsonResponse;

    using (var reader = new StreamReader(response.GetResponseStream()))
    {
         jsonResponse = reader.ReadToEnd();
    }
    return jsonResponse;
}

In my current code, the exception is thrown when the request is actually made: request.GetResponse().
What I need, essentially, is to get the JSON from the API.

Upvotes: 1

Views: 1134

Answers (3)

Jimi
Jimi

Reputation: 32248

Set SecurityProtocolType.Tls12 before you initalize the request: 

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12` 
var request = WebRequest.CreateHttp(url);

If you're on Windows 7. On Windows 10, you should only need SecurityProtocolType.SystemDefault.

Note: To enable TLS1.3 (it's available in both Windows 7 and Windows 10), If you don't use .Net 4.8 or .Net Core 3.0, since there's no enumerator for it, you can set it with: 

var secProtoTls13 = (SecurityProtocolType)12288;

Remove all the other SecurityProtocolType you have set there.

Setting the User-Agent header is also mandatory, otherwise you will receive a 404 (not found). Here's the FireFox header: 

request.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0";

A note on the User-Agent header: this specific site doesn't activate HTTP Strict Transport Security (HSTS). But some sites do, when they see that the WebBrowser supports it. HttpWebRequest doesn't understand it, so it will simply wait for a response that never comes, since the Site is waiting for interaction.
You may want to use the IE11 header instead.

Also add this other header: 

 request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");

Upvotes: 1

mjwills
mjwills

Reputation: 23898

The server side appears to be checking the user agent (presumably to stop bots and other code (like yours!) from hitting the endpoint). To bypass this, you will need to set the user agent to a value such that it thinks you are a web browser.

request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36";

works, as an example.

You may wish to consider setting the ServicePointManager.SecurityProtocol just once (at app startup) rather than on each request.

Upvotes: 1

Matt Evans
Matt Evans

Reputation: 7575

This works using HttpClient (.net 4.5 and up)

    var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
    var client = new HttpClient();
    client.DefaultRequestHeaders.Add("User-Agent", "C# App");
    Task<string> response =  client.GetStringAsync(url);
    Console.WriteLine(response.Result);

Think the server requires a user agent.

Upvotes: 0

Related Questions