CODEWITHSUNDEEP
javahttpsproxyapache-httpclient-4.x
user2209017
user2209017

Reputation: 21

Proxy-Authorization header is removed when using https

I am trying to access a https url through a proxy that requires authentication in java, and the Proxy-Authorization header is not being passed to the proxy.

I know the networking aspect is working because I can perform exactly what I need using curl:

$ curl -H "Proxy-Authorization: Basic ##########" -x my_proxy_host:80 my_https_url -v

My code seems to work when I access an http url, however when I try to access a https url I get a 403 Forbidden, and I see in the logs that the Proxy-Authorization header is not passed from Java to the proxy.

Here is my code:

public static void main(String args[]) {
    try {

        HttpHost proxy = new HttpHost(my_proxy_host, 80, "http");

        DefaultHttpClient cli = new DefaultHttpClient();

        cli.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);

        HttpHost target = new HttpHost(my_https_endpoint, 443, "https");

        HttpGet get = new HttpGet("/");

        get.setHeader("Proxy-Authorization", "Basic ##############");

        HttpResponse response = cli.execute(target, get);

        HttpEntity entity = response.getEntity();

        System.out.println("----------------------------------------");
        System.out.println(response.getStatusLine());
        if (entity != null) {
            System.out.println("Response content length: " + entity.getContentLength());
        }

        EntityUtils.consume(entity);                        
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Note if you change the following line the code above works:

HttpHost target = new HttpHost(my_https_endpoint, 80, "http");

Any ideas would be greatly appreciated.

Thanks!

Here are the logs that the apache httpclient generates.

Here is the version that doesn't work, trying to access https endpoint on port 443

09:43:43.574 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Get connection for route HttpRoute[{tls}->my_proxy_host:80->my_https_endpoint:443]
09:43:43.653 [main] DEBUG o.a.h.i.c.DefaultClientConnectionOperator - Connecting to my_proxy_host:80
09:43:43.716 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
09:43:43.716 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: CONNECT my_https_endpoint:443 HTTP/1.1
09:43:43.717 [main] DEBUG org.apache.http.wire - >> "CONNECT my_https_endpoint:443 HTTP/1.1[\r][\n]"
09:43:43.720 [main] DEBUG org.apache.http.wire - >> "Host: my_https_endpoint:443[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.wire - >> "Proxy-Connection: Keep-Alive[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.wire - >> "User-Agent: Apache-HttpClient/4.1.2 (java 1.5)[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.wire - >> "[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.headers - >> CONNECT my_https_endpoint:443 HTTP/1.1
09:43:43.721 [main] DEBUG org.apache.http.headers - >> Host: my_https_endpoint:443
09:43:43.721 [main] DEBUG org.apache.http.headers - >> Proxy-Connection: Keep-Alive
09:43:43.721 [main] DEBUG org.apache.http.headers - >> User-Agent: Apache-HttpClient/4.1.2 (java 1.5)
09:43:43.762 [main] DEBUG org.apache.http.wire - << "HTTP/1.1 403 Forbidden[\r][\n]"
09:43:43.766 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Receiving response: HTTP/1.1 403 Forbidden
09:43:43.766 [main] DEBUG org.apache.http.headers - << HTTP/1.1 403 Forbidden
09:43:43.768 [main] DEBUG o.a.h.c.p.ResponseProcessCookies - Cookie spec not specified in HTTP context
09:43:43.772 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Connection closed
09:43:43.773 [main] DEBUG o.a.h.impl.client.DefaultHttpClient - CONNECT refused by proxy: HTTP/1.1 403 Forbidden
09:43:43.773 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter@49869a03
HTTP/1.1 403 Forbidden
Response content length: 0

And here is the version that works, accessing a http endpoint through port 80 (obviously)

09:34:11.510 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Get connection for route HttpRoute[{}->my_proxy_host:80->my_http_endpoint:80]
09:34:11.646 [main] DEBUG o.a.h.i.c.DefaultClientConnectionOperator - Connecting to my_proxy_host:80
09:34:11.720 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: best-match
09:34:11.742 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
09:34:11.743 [main] DEBUG o.a.h.impl.client.DefaultHttpClient - Attempt 1 to execute request
09:34:11.743 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: GET my_http_endpoint:80/ HTTP/1.1
09:34:11.744 [main] DEBUG org.apache.http.wire - >> "GET my_http_endpoint:80/ HTTP/1.1[\r][\n]"
09:34:11.746 [main] DEBUG org.apache.http.wire - >> "Proxy-Authorization: Basic ###################[\r][\n]"
09:34:11.746 [main] DEBUG org.apache.http.wire - >> "Host: my_http_endpoint:80[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.wire - >> "Proxy-Connection: Keep-Alive[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.wire - >> "User-Agent: Apache-HttpClient/4.1.2 (java 1.5)[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.wire - >> "[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.headers - >> GET my_http_endpoint:80/ HTTP/1.1
09:34:11.747 [main] DEBUG org.apache.http.headers - >> Proxy-Authorization: Basic ###################
09:34:11.747 [main] DEBUG org.apache.http.headers - >> Host: my_http_endpoint:80
09:34:11.747 [main] DEBUG org.apache.http.headers - >> Proxy-Connection: Keep-Alive
09:34:11.747 [main] DEBUG org.apache.http.headers - >> User-Agent: Apache-HttpClient/4.1.2 (java 1.5)
09:34:11.880 [main] DEBUG org.apache.http.wire - << "HTTP/1.1 200 OK[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Date: Fri, 29 Mar 2013 14:34:10 GMT[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Expires: -1[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Cache-Control: private, max-age=0[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Content-Type: text/html; charset=ISO-8859-1[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Set-Cookie: PREF=ID=918523062b63f55d:FF=0:TM=1364567650:LM=1364567650:S=T8IIsiH2_cw1UMLI; expires=Sun, 29-Mar-2015 14:34:10 GMT; path=/; domain=.google.com[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Set-Cookie: NID=67=OHgx0xi1JDegF7uPjPUfW_pCC5Yn0H9S8-CMxyKbtgrF2vPQ-svFv1G4h5yxwCVEi2gDA23tywtEiYQvbPCHxDpAkefUlDJrWK94fEmXaiwuvua5w50eAej0yy3ysI4N; expires=Sat, 28-Sep-2013 14:34:10 GMT; path=/; domain=.google.com; HttpOnly[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "P3p: CP="This is not a P3P policy! See my_http_endpoint/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Server: gws[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "X-Xss-protection: 1; mode=block[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "X-Frame-options: SAMEORIGIN[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Transfer-Encoding: chunked[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Connection: close[\r][\n]"
09:34:11.891 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
09:34:11.892 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
09:34:11.892 [main] DEBUG org.apache.http.headers - << HTTP/1.1 200 OK
09:34:11.892 [main] DEBUG org.apache.http.headers - << Date: Fri, 29 Mar 2013 14:34:10 GMT
09:34:11.892 [main] DEBUG org.apache.http.headers - << Expires: -1
09:34:11.892 [main] DEBUG org.apache.http.headers - << Cache-Control: private, max-age=0
09:34:11.892 [main] DEBUG org.apache.http.headers - << Content-Type: text/html; charset=ISO-8859-1
09:34:11.892 [main] DEBUG org.apache.http.headers - << Set-Cookie: PREF=ID=918523062b63f55d:FF=0:TM=1364567650:LM=1364567650:S=T8IIsiH2_cw1UMLI; expires=Sun, 29-Mar-2015 14:34:10 GMT; path=/; domain=.google.com
09:34:11.892 [main] DEBUG org.apache.http.headers - << Set-Cookie: NID=67=OHgx0xi1JDegF7uPjPUfW_pCC5Yn0H9S8-CMxyKbtgrF2vPQ-svFv1G4h5yxwCVEi2gDA23tywtEiYQvbPCHxDpAkefUlDJrWK94fEmXaiwuvua5w50eAej0yy3ysI4N; expires=Sat, 28-Sep-2013 14:34:10 GMT; path=/; domain=.google.com; HttpOnly
09:34:11.892 [main] DEBUG org.apache.http.headers - << P3p: CP="This is not a P3P policy! See my_http_endpoint/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
09:34:11.892 [main] DEBUG org.apache.http.headers - << Server: gws
09:34:11.892 [main] DEBUG org.apache.http.headers - << X-Xss-protection: 1; mode=block
09:34:11.892 [main] DEBUG org.apache.http.headers - << X-Frame-options: SAMEORIGIN
09:34:11.892 [main] DEBUG org.apache.http.headers - << Transfer-Encoding: chunked
09:34:11.892 [main] DEBUG org.apache.http.headers - << Connection: close
09:34:11.916 [main] DEBUG o.a.h.c.p.ResponseProcessCookies - Cookie accepted: "[version: 0][name: PREF][value: ID=918523062b63f55d:FF=0:TM=1364567650:LM=1364567650:S=T8IIsiH2_cw1UMLI][domain: .google.com][path: /][expiry: Sun Mar 29 09:34:10 CDT 2015]"
09:34:11.917 [main] DEBUG o.a.h.c.p.ResponseProcessCookies - Cookie accepted: "[version: 0][name: NID][value: 67=OHgx0xi1JDegF7uPjPUfW_pCC5Yn0H9S8-CMxyKbtgrF2vPQ-svFv1G4h5yxwCVEi2gDA23tywtEiYQvbPCHxDpAkefUlDJrWK94fEmXaiwuvua5w50eAej0yy3ysI4N][domain: .google.com][path: /][expiry: Sat Sep 28 09:34:10 CDT 2013]". 
HTTP/1.1 200 OK
Response content length: -1
09:34:11.925 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter@303459ed
09:34:11.925 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Released connection open but not reusable.
09:34:11.926 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Connection shut down

Upvotes: 2

Views: 11463

Answers (2)

user2209017
user2209017

Reputation: 21

I solved my problem by creating my own proxy on amazon EC2.

My goal was to be able to access an https service that required IP whitelisting, from an application running on Heroku, so there is no static IP or IP range that I can whitelist, therefore I needed a proxy with a static IP.

I was trying to use the proximo heroku addon and was having the problem I described above.

Thanks to Dave G for the help!

Upvotes: 0

Dave G
Dave G

Reputation: 9767

Try this variant and see if it might meet your needs:

public static void main(String args[]) {
    try {

        HttpHost proxy = new HttpHost(my_proxy_host, 80, "http");

        DefaultHttpClient cli = new DefaultHttpClient();

        cli.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
        cli.getCredentialsProvider().setCredentials(
            new AuthScope(proxy.getHostName(), proxy.getPort()),
            new UsernamePasswordCredentials("username", "password"));

        HttpHost target = new HttpHost(my_https_endpoint, 443, "https");

        HttpGet get = new HttpGet("/");

        // Example from: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html#d5e1032
        BasicAuthCache authCache= new BasicAuthCache();
        // Target the proxy not target host
        authCache.put(proxy, new BasicScheme(ChallengeState.PROXY));
        BasicHttpContext context= new BasicHttpContext();
        context.setAttribute(ClientContext.AUTH_CACHE, authCache);
        HttpResponse response = cli.execute(target, get, context);

        HttpEntity entity = response.getEntity();

        System.out.println("----------------------------------------");
        System.out.println(response.getStatusLine());
        if (entity != null) {
            System.out.println("Response content length: " + entity.getContentLength());
        }

        EntityUtils.consume(entity);                        
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Ideally, the proxy should have responded with an HTTP 407 Proxy Authentication Required on the initial request instead of a 403 Forbidden. The 403 basically is saying GO-AWAY! While the 407 says, "hey - you wanna come through? give me a reason why?" by responding with a "Proxy-Authenticate: " header, to which you must respond with your credentials via a "Proxy-Authorization: " header.

By preemptively authenticating, you're essentially sending credentials to a host when they have not been asked, kind of like stapling your Social Security number in large letters to your forehead.

EDIT I think I may have found something that MIGHT get you over the hump on this one: HttpClient 4.2.2 and proxy with username/password see if that kicks it into gear.

Upvotes: 3

Related Questions