How to get an HTTP Proxy response Header in Golang built-in HTTP client?

Question

I'm doing a simple HTTP GET request with the default net/http Golang lib via HTTP proxy and want to read the content of the first proxy reply(for the HTTP client req with the CONNECT method).

In plain text, it looks like this

HTTP/1.1 200 OK
Request-Uid: <some id>
<another header>: <another value>

Golang code:

...
proxyUrlParsed, errUrl := url.Parse(proxyUrl)
tr := &http.Transport{
   Proxy:   http.ProxyURL(proxyUrlParsed),
}
client := &http.Client{
   Transport: tr,
}
request, errReq := http.NewRequest("GET", targetUrl, nil)
response, errDo := client.Do(request)
// Response contains HTTP headers from the reply from the 
// target resource but not the intermediate proxy.

I partially solved it with DialContext, but I were needed to impl some parts of the protocol that I found not so handy and costly for later support. So is there an easy and clever way to do it?

Answer 1

In Golang >=1.20 this task can be solved using standard library https://pkg.go.dev/net/http@master#Transport.OnProxyConnectResponse

Code example https://github.com/greggyNapalm/proxychick/blob/v0.0.1/pkg/httpx/httpx.go#L116

Answer 2

tunnel proxy

Take the use of the curl client as an example. When requesting https, use the CONNECT method to connect to the tunnel connection. The obtained stream content is TLS encrypted content, which cannot be decrypted by the proxy.

If have a tls certificate, can try to parse the response stream.

When wireshark captures https requests, a parameter needs to be configured in the browser. The certificate is saved in the specified file

http_proxy=127.0.0.1:8021 https_proxy=127.0.0.1:8021 curl -v https://qq.com
http_proxy=127.0.0.1:8021 https_proxy=127.0.0.1:8021 curl -v https://qq.com

package main

import (
    "fmt"
    "io"
    "log"
    "net"
    "net/http"
    "net/http/httputil"
    "os"
)

func main() {
    proxy := func(w http.ResponseWriter, req *http.Request) {
        log.Println("proxy", req.Method, req.RequestURI)
        if req.URL.Host != "" {
            if req.Method == http.MethodConnect {
                // tunnel
                conn, err := net.Dial("tcp", req.URL.Host)
                if err != nil {
                    w.WriteHeader(502)
                    fmt.Fprint(w, err)
                    return
                }

                client, _, err := w.(http.Hijacker).Hijack()
                if err != nil {
                    w.WriteHeader(502)
                    fmt.Fprint(w, err)
                    conn.Close()
                    return
                }
                client.Write([]byte("HTTP/1.0 200 OK\r\n\r\n"))

                hr, hw := io.Pipe()
                go func(){
                    io.Copy(os.Stdout, hr)
                    hr.Close()
                }()
                go func() {
                    // print response to stdout
                    io.Copy(io.MultiWriter(client, hw), conn)
                    client.Close()
                    conn.Close()
                    hw.Close()
                }()
                go func() {
                    io.Copy(conn, client)
                    client.Close()
                    conn.Close()
                }()
                return
            }

            httputil.NewSingleHostReverseProxy(req.URL).ServeHTTP(w, req)
        }
    }
    http.ListenAndServe(":8021", http.HandlerFunc(proxy))
}

reverse proxy

use /net/http/httputil.ReverseProxy proxy a request ,set ModifyResponse field is response hook.

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    proxy := httputil.NewSingleHostReverseProxy(&url.URL{
                                                Scheme: "http", 
                                                Host: "127.0.0.1:8020"})

    proxy.ModifyResponse = func(w *http.Response) error {
        w.Header.Add("Author", "eudore")
        log.Println(w.Request.Method, w.Request.RequestURI, w.Status)
        return nil
    }
    http.ListenAndServe(":8021", proxy)
}

Curl requests

With port 8020 like [root@node1 ~]# curl -I 127.0.0.1:8020?222

HTTP/1.1 401 Unauthorized
Www-Authenticate: Basic
Date: Thu, 17 Nov 2022 01:34:06 GMT

Or port 8021 like [root@node1 ~]# curl -I 127.0.0.1:8021?222

HTTP/1.1 401 Unauthorized
Author: eudore
Date: Thu, 17 Nov 2022 01:34:07 GMT
Www-Authenticate: Basic