Java HttpURLConnection | POST method | Sequence of HTTP header fields

Question

I am trying to "spoof" a Firefox HTTP POST request in Java using java.net.HttpURLConnection. I use Wireshark to check the HTTP headers being sent, so I have (hopefully) reliable source of information, why the Java result doesn't match the ideal situation (using Firefox).

I have set all header fields exactly to the values that Firefox sends via HTTP and noticed, that the sequence of the header fields is not the same. The output for Firefox is like:

POST ...
**Host**
User-Agent
Accept
Accept-Language
Accept-Encoding
Referer
Connection
Content-Type
Content-Length

When I let wireshark tap off my implementation in Java, it gives me a slightly different sequence of fields:

POST...
**User-Agent**
Accept
Accept-Language
Accept-Encoding
Referer
Content-Type
Host
Connection
Content-Length

So basically, I have all the fields, just in a different order. I have also noticed that the Host field is sent with a different value: www.thewebsite.com (Firefox) <---> thewebsite.com (Java HttpURLConnection), although I pass on the String to httpUrlConnection.setRequestProperty with the "www."

I have not yet analyzed the byte output of Wireshark, but I know that the server is not returning the same Location in the header fields of my response.

My questions are:

(1) Is is possible to control the sequence the header fields in the request, and if yes is it possible to do using HttpURLConnection? If not, is it possible to directly control the bytes in the HTTP header using Java? [I don't own the server, so my only hope to get the POST method working is through my application pretending to be Firefox, the server is not really verbose, my only info are: Apache with PHP]

(2) Is there a way to fix the setRequestProperty() problem ("www") as described above?

(3) What else could matter? (Do I need to concern the underlying layers, TCP....?)

Thanks for any comments.

PS. I am trying to model a situation without cookies being sent, so that I can ignore the effect.

Answer 1

I got lucky with Apache Http Components, my guess is that the "Host" header's missing "www." made the difference, which can be set exactly as intended using Apache's HttpPost:

httpPost.setHeader("Host", "www.thewebsite.com");

The Wireshark output confirmed my suspicion. Also this time the TCP communication prior to my HTTP post looks different (client ---> server, server ---> client, client ---> server) instead of (client ---> server, server ---> client, client ---> server, client---> server).

Now I get the desired Location header value and the server is also setting the cookies. :)

For the most part, this question is resolved.

Actually I wanted to use the lightweihgt HttpUrlConnection because that's what the Android Developers blog suggesting. The System.setProperty("sun.net.http.allowRestrictedHeaders", "true") might work as well, if it allows to "www." in the Host value.

Answer 2

the headers order is not important. the headers got by server are also out-of-order. And you can not control httpUrlConnection header order. But if you write your own TCP client, you can control your header order. like:

clientSocket = new Socket(serverHost, serverPort);

OutputStream os = clientSocket.getOutputStream();

String send = "GET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\nGET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\n";

os.write(send.getBytes());

The Second question is answered by Marcel Stör in the first answer. a

Answer 3

First, the order of the headers is irrelevant.

Second, in order to manually override the host header you need to set sun.net.http.allowRestrictedHeaders=true either in code

System.setProperty("sun.net.http.allowRestrictedHeaders", "true")

or at JVM start

-Dsun.net.http.allowRestrictedHeaders=true

This is a security precaution introduced by Oracle a while ago. That's because according to RFC

The Host request-header field specifies the Internet host and port number of the resource being requested, as obtained from the original URI given by the user or referring resource (generally an HTTP URL).