CODEWITHSUNDEEP
httpurlconnectionchunked
Tanya
Tanya

Reputation: 203

How to make sure that you have read the "full message" when using "Transfer-Encoding:chunked" http response

I am using HttpUrlConnection (java) to read the http chunked response(Transfer-Encoding:chunked) like following and I am able to read the message. But, how can I make sure that I have read all the chunks the correctly and the message read is intact..

BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()));
        String inputLine;
        StringBuffer response = new StringBuffer();

        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();

Upvotes: 3

Views: 1414

Answers (1)

William Rosenbloom
William Rosenbloom

Reputation: 2626

Hey I'm sorry for the late response but, as you might guess, I only just came across your question.

You seem to have a small misunderstanding of what chunked transfer encoding means. It does not mean that the message will be sent in CRLF ended lines. That seems to be what you're trying to implement, but it's a little more complicated than that.

Incase you're not familiar with what a CRLF is, it just means Carriage-Return Line-Feed (or \r\n in traditional Unix escape sequences).

Chunked transfer encoding means that the message will be sent in chunks not lines. Each chunk has a format that is even more self explanatory than a line ending in a CRLF.

How Chunks Work

Each chunk begins with a sequence of characters in the range [0-9, 'a'-'f']. This is a hexadecimal number representing the length, in bytes, of the chunk. This hexadecimal number will be followed by a CRLF. This will be followed by the chunk data (hopefully with the specified length). The chunk data will be ended with another CRLF.

You will get several of these chunks sequentially, and it is your job to concatenate them in the order in which they arrived. You read them in order until you get a special chunk that signals the end of the message body. The only thing special about this last chunk is that it will have a length of zero and contain no data, i.e. the chunk will be 0\r\n\r\n.

Unfortunately Java's HttpURLConnection class only implements automatic chunked transfer encoding, and you have to do the decoding yourself.

Cumbersome as it might be, if you implement this protocol you can be confident about whether you have read the entire message...with one caveat...

You Might Have to Look for Trailers

Trailers are the same as headers except they come at the end of an HTTP message, whereas headers come at the beginning. Chunked transfer encoding will allow you to find the end of a message body but the full HTTP message might continue.

The HttpURLConnection class will not parse trailers for you. The class parses the HTTP head (status line and headers) before directing its socket (or whatever it uses) to the connection's OutputStream. Once the socket is directed toward that OutputStream it doesn't go back. It's up to you to process the data from then on.

Luckily trailers are much more predictable than headers...or at least they are supposed to be. Every trailer the server will send is supposed to be specified in a semicolon separated list in the Trailers header field, i.e. Trailers: Content-disposition; Cache-control; From in the headers means you should look for Content-disposition, Cache-control, and From trailer fields after the message.

Trailers are allowed only in chunked encoded messages, and they follow the same format as headers: field name, colon, space, field value, CRLF. Just like with headers, if a trailer is followed by two CRLF's instead of one that means it was the last trailer.

Upvotes: 2

Related Questions