Unable to connect with HttpsUrlConnection but able to connect using HttpClient

Question

I am trying to convert from HttpClient to HttpsUrlConnection but when I try to connect using HttpsUrlConnection I get a 403 response code. The HttpClient code works perfectly. I do not know what to do to get around the error and connect successfully using HttpsUrlConnection. I've tried using TrustManager allowing all certificates which did not work and would not want to use in production code. Here is the original HttpClient code:

            HostnameVerifier hostnameVerifier = 
                org.apache.http.conn.ssl.SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER;

            HttpClient httpClient = new DefaultHttpClient();
            HttpParams params = httpClient.getParams();
            HttpConnectionParams.setConnectionTimeout(params, 10000);
            HttpConnectionParams.setSoTimeout(params, 10000);

            SchemeRegistry registry = new SchemeRegistry();

            SSLSocketFactory sSSocketFactory = SSLSocketFactory.getSocketFactory();
            sSSocketFactory.setHostnameVerifier((X509HostnameVerifier) hostnameVerifier);
            registry.register(new Scheme("https", sSSocketFactory, 443));

            PlainSocketFactory pSocketFactory = PlainSocketFactory.getSocketFactory();
            registry.register(new Scheme("http", pSocketFactory, 80));

            SingleClientConnManager mgr = new SingleClientConnManager(httpClient.getParams(), registry);
            httpClient = new DefaultHttpClient(mgr, httpClient.getParams());

            HttpContext localContext = new BasicHttpContext();
            InputStream instream = null;
            String url = "";
            String tagOpen = "";
            String tagClose = "";

            url = "https://example.com";
            
            HttpGet httpGet = new HttpGet(url);

            HttpResponse response = null;
           
            try {
                response = httpClient.execute(httpGet, localContext);
                int responseCode = response.getStatusLine().getStatusCode();
                HttpEntity entity = response.getEntity();
                if (entity != null) {
                    instream = entity.getContent();
                    int r = -1;
                    StringBuffer respStr = new StringBuffer();
                    while ((r = instream.read()) != -1)
                        respStr.append((char) r);

                    responseString = respStr.toString();
                 

                    instream.close();
                }
            } catch (Exception e) {
                
            } finally {
                if (instream != null) {
                    try {
                        instream.close();
                    } catch (Exception e) {
                    }
                }
            }

And here is the HttpsUrlConnection code that fails with a 403 response code:

InputStream instream = null;
String url = "";

url = "https://example.com";

HttpsURLConnection httpsURLConnection = null;

try {
    URL requestUrl = new URL(url);

    httpsURLConnection = (HttpsURLConnection)   requestUrl.openConnection();
    httpsURLConnection.setReadTimeout(10000);
    httpsURLConnection.setConnectTimeout(10000);


    HostnameVerifier hostnameVerifier = org.apache.http.conn.ssl.SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER;



    httpsURLConnection.setHostnameVerifier(hostnameVerifier);

    httpsURLConnection.setSSLSocketFactory((javax.net.ssl.SSLSocketFactory) javax.net.ssl.SSLSocketFactory.getDefault());

    httpsURLConnection.connect();
    int responseCode = httpsURLConnection.getResponseCode();


    instream = httpsURLConnection.getInputStream();

    if (instream != null) {
        int r = -1;
        StringBuffer respStr = new StringBuffer();
        while ((r = instream.read()) != -1)
            respStr.append((char) r);

        responseString = respStr.toString();
      

        instream.close();
    }
} catch (Exception e) {
    
} finally {
    if (instream != null) {
        try {
            instream.close();
        } catch (Exception e) {
        }
    }
    if(httpsURLConnection != null){
        httpsURLConnection.disconnect();
    }
}

Thanks for your help!

Answer 1

The server needed a user agent and the problem was fixed with this line of code:

httpsURLConnection.setRequestProperty("User-Agent", "user-agent-example");