CODEWITHSUNDEEP
go
Jérôme R
Jérôme R

Reputation: 1257

Reading gzipped HTTP response in Go

I am trying to read a gzipped HTTP response with Go! but I always get the following error message :

panic: gzip: invalid header
[...] stack trace [...]

If I run "curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -" I get the response correctly gunzipped. I also double checked with ngrep and the pair Accept-Encoding/Content-Encoding is correctly sent/returned.

If I create a file with some dummy content and gzip it, I can read it from my Go! program.

The program I used for testing:

package main

import (
    "io"
    //"os"
    "fmt"
    "compress/gzip"
    "net/http"
)

func main() {
    /* This works fine
    f, _ := os.Open("/tmp/test.gz")
    defer f.Close()
    reader, err := gzip.NewReader(f)
    */

    // This does not :/
    resp, _ := http.Get("http://foo.com/")
    defer resp.Body.Close()
    reader, err := gzip.NewReader(resp.Body)

    if err != nil { panic(err) }

    buff := make([]byte, 1024)
    for {
        n, err := reader.Read(buff)

        if err != nil && err != io.EOF {
            panic(err)
        }

        if n == 0 {
            break
        }
    }

    s := fmt.Sprintf("%s", buff)
    fmt.Println(s)
}

Have I overlooked something ?

Upvotes: 41

Views: 54765

Answers (4)

Clare Chu
Clare Chu

Reputation: 778

The following is an example of manually handling compression.


import "compress/gzip"

func ReadAll(r io.Reader) ([]byte, error) {
    reader, err := gzip.NewReader(r)
    if err != nil {
        return nil, err
    }
    defer reader.Close()
    buff, err := ioutil.ReadAll(reader)
    return buff, err
}

Upvotes: 0

simonmenke
simonmenke

Reputation: 2880

net/http#Transport handles gzip compressed responses. You don't have to do anything special.

Take a look at the DisableCompression option, here.

Upvotes: 41

Requilence
Requilence

Reputation: 241

According to net/http docs (line 110) if you manually set the Accept-Encoding request header, than gzipped response will not automatically decompressed by the http.Transport. Otherwise, behavior is controlled by the Transport's DisableCompression boolean

Upvotes: 24

Stephen Weinberg
Stephen Weinberg

Reputation: 53408

EDIT: The following is an example of manually handling compression. If you don't set the header, the default Transport will do it for you and then decompress while you read the response.Body.

client := new(http.Client)

request, err := http.NewRequest("GET", "http://stackoverflow.com", nil)
request.Header.Add("Accept-Encoding", "gzip")

response, err := client.Do(request)
defer response.Body.Close()

// Check that the server actually sent compressed data
var reader io.ReadCloser
switch response.Header.Get("Content-Encoding") {
case "gzip":
    reader, err = gzip.NewReader(response.Body)
    defer reader.Close()
default:
    reader = response.Body
}

io.Copy(os.Stdout, reader) // print html to standard out

Error handling removed for brevity. I kept the defers.

Upvotes: 72

Related Questions