CODEWITHSUNDEEP
gogzipreverse-proxy
code
code

Reputation: 61

Golang DumpResponse gzip issue when logging response body (reverseproxy)

I have created a simple reverse proxy in my main.go as follows:

reverseproxy := httputil.NewSingleHostReverseProxy("https://someurl/someuri")

this is working fine, however, I would like to log the response that comes back from the server. I can also do this utilizing the following code inside the standard RoundTrip method recommended by golang:

response, err := http.DefaultTransport.RoundTrip(request)
dumpresp, err := httputil.DumpResponse(response, true)

if err != nil {
    return nil, err
}
log.Printf("%s", dumpresp)

All the above works as expected aside from one thing, the response, when Content-Encoding: gzip, the string appears to the logs as non-utf8 gzip characters. it seems to be a golang oversight but maybe there is something i have missed in the documentation, which I have read to its completion a few times. I can't post the logs here because the characters are non utf8 so they would not display on this site anyway. So, I know what your thinking, just grab the gzip content and use a method to remove the gzip compression. That would be great if the response from DumpResponse was not partly gzip and partly standard utf8 with no way to separate the sections from each other.

So I know what your going to say, why not just take the raw response like the following and gzip decode, and "not" use DumpResponse. Well I can do that as well, but there is an issue with the following:

var reader io.Reader
startreader := httputility.NewChunkedReader(reader)
switch response.Header.Get("Content-Encoding") {
case "gzip":
    startreader, err = gzip.NewReader(response.Body)
    log.Println("Response body gzip: ")
    buf := new(bytes.Buffer)
    buf.ReadFrom(startreader)
    b := buf.Bytes()
    s := *(*string)(unsafe.Pointer(&b))
    for name, value := range response.Header {
        var hvaluecomp string = ""
        for i := 0; i < len(value); i++ {
            hvaluecomp += value[i]
        }
        response.Header.Add(name,hvaluecomp)
    }
    log.Printf("%s", s)



default:
    startreader = response.Body
    buf := new(bytes.Buffer)
    buf.ReadFrom(startreader)
    b := buf.Bytes()
    s := *(*string)(unsafe.Pointer(&b))
    for name, value := range response.Header {
        fmt.Printf("%v: %v\n", name, value)
    }
    log.Printf("%s", s)
}

The issue with the above is, responses can only be read one time via the reader, after that the response cannot be read by the reverse proxy anymore and it makes the proxy response back to the browser nil =), so again I was met with failure. I cant imagine that the folks coding golang would have missed decoding gzip, just strange, it seems to be so simple.

So in the end, my question is, does DumpResponse give me the ability to decompress gzip so I can log the actual response instead of non utf8 characters? This does the log reader no good when debugging an issue for the production product. This would render the built in golang reverse proxy useless in my eyes and I would start development on my own.

Upvotes: 0

Views: 1311

Answers (2)

code
code

Reputation: 61

The answer is to copy the stream, then you can use the second variable to re-post the stream back to the request.body object so you don't lose any data.

buf, _ := ioutil.ReadAll(response.Body)
responseuse1 := ioutil.NopCloser(bytes.NewBuffer(buf))
responsehold := ioutil.NopCloser(bytes.NewBuffer(buf))

Method to pull logging: extractLogging(responseuse1) Push the hold back to the body so its untouched: response.Body = responsehold

return response

Upvotes: 1

Volker
Volker

Reputation: 42413

does DumpResponse give me the ability to decompress gzip

No it does not.

To me it seems as if the major problem is neither the reverse proxy nor DumpResponse but that you are trying to "log" binary data: Be it gzip, or other binary data like images. Just fix your logging logic: If the raw body is binary you should render some kind of representation or transformation of it. For gziped stuff gunzip it first (but this might still be binary "non utf8" data unsuitable for logging). Focus on the real problem: How to "log" binary data.

Upvotes: 0

Related Questions