CODEWITHSUNDEEP
httpgogetcontent-length
Jordan
Jordan

Reputation: 919

Unexpected http/net response content length in Golang

I am making a HTTP Get request using the code below. I proxy it through Fiddler to analyse the request. The request seems to be being made correctly, with the expected response. However, the resp.ContentLength property of the response in GO is -1 every time despite the fact that the response in Fiddler shows a positive integer value of Content-Length: 1830.

Why is the ContentLength not being picked up in GO?

Go Code

package main

import "net/http"
import "os"
import "fmt"

func main() {
    os.Setenv("HTTP_PROXY", "http://127.0.0.1:8888") // For Fiddler debugging
    resp,err := http.Get("http://www.google.com/robots.txt")
    if (err == nil) {
        if (resp.StatusCode == 200) {
            fmt.Println(resp.ContentLength) // Prints -1
        }
    }
}

Resulting Request in Fiddler

GET /robots.txt HTTP/1.1
Host: www.google.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

Response Received in Fiddler

HTTP/1.1 200 OK
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Type: text/plain
Content-Length: 6907
Date: Mon, 05 Mar 2018 13:53:00 GMT
Expires: Mon, 05 Mar 2018 13:53:00 GMT
Cache-Control: private, max-age=0
Last-Modified: Thu, 01 Mar 2018 18:00:00 GMT
X-Content-Type-Options: nosniff
Server: sffe
X-XSS-Protection: 1; mode=block

Upvotes: 3

Views: 5740

Answers (3)

Alan Gus
Alan Gus

Reputation: 1

The content length change in func readTransfer https://github.com/golang/go/blob/master/src/net/http/transfer.go

Upvotes: 0

thks173
thks173

Reputation: 1728

Just want to share my research.

There's a DisableCompression option in the http.Transport struct, which default to false (http.Get use this default option). With that option, go's internal package will forcing gzip compression by adding the Accept-Encoding: gzip header in the request. After receiving the response, go's internal package will do the decompression job for us, so that we don't have to deal with the gzip chunk. They also delete the original Content-Length header because it's not correct anymore.

All of that "magic under the hood" behaviors give us what we see here, missing the Content-Length header.

Ref: https://github.com/golang/go/blob/master/src/net/http/transport.go#L182

Upvotes: 2

Roman Kiselenko
Roman Kiselenko

Reputation: 44360

The value -1 indicates that the length is unknown. (in most cases this indicates chunked encoding).

Upvotes: 11

Related Questions