CODEWITHSUNDEEP
c#asp.net
Johan
Johan

Reputation: 35194

Inspect headers set by WebClient proxy

I'm executing request through some free proxy servers, and I would like to know what headers each proxy server sets. Right now I'm visiting a page that prints out the result in the html body.

using(WebClient client = new WebClient())
{
    WebProxy wp = new WebProxy("proxy url");
    client.Proxy = wp;
    string str = client
                  .DownloadString("http://www.pagethatprintsrequestheaders.com");
}

The WebClient doesn't show the modified headers, but the page prints the correct ones. Is there any way to find out what headers that are being set by the proxy without visiting a page that prints them like in my example? Do I have to create my own http listener?

Upvotes: 1

Views: 514

Answers (2)

Grhm
Grhm

Reputation: 6834

It looks like your sending a request from your WebClient, through the proxy and its received by the host at www.pagethatprintsrequestheaders.com.

If the proxy is adding headers to the request, your webclient will never see them on it's request.

        webclient             proxys request 
         request            with headers added
client -----------> proxy ----------------------> destination host

The webclient can only see the state of the request between it and the proxy. The proxy will create a new request to send to the destination host, and its that request to which the headers are added. It also that request that is received by the destination host (which is why when it echoes back the headers it can see those added by the proxy)

When the response comes back, the headers are set by the host. It's possible that the proxy will add some headers to the response, but even if it did, they are not likely to be the same headers it adds to a request.

            response                      response
        (forwarded by proxy)        (headers set by host)
client <------------------- proxy <------------------------- destination host

Using a host that echo the headers back as part of the response payload is one option.
Another would be to use something between the proxy and the destination host to inspect the request there (e.g a packet sniffer or another proxy like Fiddler that lets you see the request headers).

If the proxy is outside of you network, getting between the proxy and the destination host will be difficult (unless the host is under your control).

Upvotes: 2

Mendhak
Mendhak

Reputation: 8775

When the proxy server sets its own headers, it is essentially performing its own web request. It can even hide or override some of the headers that you set using your WebProxy.

Consequently, only the target page (pagethatprintsrequestheaders.com) can reliably see the headers being set by the proxy. There is no guarantee that the proxy server will send back the headers that it had sent to the target, back to you.

To put it another way, it really depends on the proxy server implementation. if the proxy server you are using is based on Apache's ProxyPass, you'd probably see the headers being set! If it's a custom implementation, then you may not see it.

You can first try inspecting the client.ResponseHeaders property of the WebClient after your response comes back. If this does not contain headers matching what (pagethatprintsrequestheaders.com) reports, then it's indeed a custom or modified implementation.

You could then create your own proxy servers, but this is more involved. You would probably spin up an EC2 instance, install Squid/TinyProxy/YourCustomProxy on it and use that in your WebProxy call.

You may also want to modify your question and explain why you want to read the headers. There may be solutions to your overall goal that don't require reading headers at all but could be done in some other way.

Upvotes: 3

Related Questions