CODEWITHSUNDEEP
.netauthenticationhttpshttpwebrequest
Jimmy
Jimmy

Reputation: 5241

Retrieving https page using .net HttpWebRequest or WebClient returns (400) Bad Request

I have an account with Interactive Brokers. They have a decent trading API, but there is no API to access some of the account management features that are available through their web site at https://www.interactivebrokers.com/sso/Login . I looked at the javascript a bit, and they are encrypting the user name & password in the browser in javascript, and then sending the encrypted stuff over SSL.

I'm hoping to be able to do that in c#. The first step (I think) is to open https://www.interactivebrokers.com/sso/Login using an HttpWebRequest or WebClient, to get a session ID (this is just loading the login page -- not even entering any username or password yet).

There's nothing too ground-breaking in the code. The WebClient version is:

WebClient client = new WebClient();
client.Encoding = Encoding.UTF8;
Stream data = client.OpenRead("https://www.interactivebrokers.com/sso/Login");

The HttpWebRequest version is nothing to write home about either:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.interactivebrokers.com/sso/Login");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

However, in either case when I do that, I get a 

The remote server returned an error: (400) Bad Request.

I tried to add a user agent, as per Why Does my HttpWebRequest Return 400 Bad request? :

request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";

and

request.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)";

I still get the (400) error.

The inner exception is null, but looking at the WebException's stream shows the following response (I don't know why the title says '200 OK'):

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>200 OK</title>
</head><body>
<h1>OK</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>

I looked at the difference in packets between a web browser doing the request and my c# code, and the first difference I noticed is that the web browser has many more cypher suites available to it then my c# code. When the server responds to the web browser, it chooses one of the cyphers that is not available in the c# client. How do I add cypher suites to HttpWebRequest / WebClient? I haven't yet figured out if that is actually the problem or not.

Am I missing something really basic? Why can't I even load that page with an HttpWebRequest or WebClient?

Thanks

Upvotes: 0

Views: 2712

Answers (1)

Jimmy
Jimmy

Reputation: 5241

It turns out that two headers (properties on HttpWebRequest) were required: UserAgent & Accept. It's something I should have spotted with WireShark, but using Fiddler made it a bit easier to compare headers. I was slowed down because I was comparing the CONNECT sessions in fiddler, and changing the useragent doesn't affect the connect headers (don't really know why, but it doesn't matter I guess). The next line down after CONNECT properly reflected the properties that I set on HttpWebRequest, and I was then able to figure out what was wrong.

Thanks to Eric Law for stirring me in the right direction. Had nothing to do with Cyphers :)

Upvotes: 1

Related Questions