Can't connect to specific https website using WebClient in Java

Question

Summary: I've created a simple block of code that I've used several times in the past with success, but it is not working with this website ("https://mid-usa.com/login") for some reason. I can telnet from my computer to mid-usa.com on 443, and I can open this page in a browser.

    WebClient client = new WebClient(BrowserVersion.BEST_SUPPORTED);
    client.getOptions().setSSLClientProtocols(new String[] { "TLSv1.2","TLSv1.1"});
    client.getOptions().setCssEnabled(false);
    client.getOptions().setJavaScriptEnabled(false);
    client.getOptions().setUseInsecureSSL(true);
    client.getOptions().setTimeout(10000);
    HtmlPage page = (HtmlPage) client.getPage("https://mid-usa.com/login");

The reason I am setting the setSSLClientProtocols to TLSv1.2, is because upon looking at the website's protocols running a scanner, I thought it would help (from reading other posts), but it didn't. I also left it with just TLSv1.2, and still get the following print out:

Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Connection resetjava.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at sun.security.ssl.InputRecord.readFully(Unknown Source)
    at sun.security.ssl.InputRecord.read(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:184)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:193)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1537)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1456)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:447)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:368)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:520)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:501)
    at ScraperBase.autoLogin(ScraperBase.java:25)
    at MidwayDataCollection$4.actionPerformed(MidwayDataCollection.java:222)
    at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
    at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
    at java.awt.Component.processMouseEvent(Unknown Source)
    at javax.swing.JComponent.processMouseEvent(Unknown Source)
    at java.awt.Component.processEvent(Unknown Source)
    at java.awt.Container.processEvent(Unknown Source)
    at java.awt.Component.dispatchEventImpl(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Window.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
    at java.awt.EventQueue.access$500(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue.dispatchEvent(Unknown Source)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.run(Unknown Source)

Any ideas?

Answer 1

I believe that the problem is that the cipher suite the server wants to use is disabled in later JDKs.

When I connected via curl -v https://mid-usa.com >/dev/null I got:

* SSL connection using TLSv1.2 / DES-CBC3-SHA

This maps to SSL_RSA_WITH_3DES_EDE_CBC_SHA in JSSE (https://docs.oracle.com/cd/E19728-01/820-2550/cipher_suites.html)

I rewrote your example to just use a URLConnection:

        URL url = new URL("https://mid-usa.com");
        URLConnection urlConnection = url.openConnection();
        InputStream is = urlConnection.getInputStream();

And ran it with -Djavax.net.debug=all -Dhttps.cipherSuites="SSL_RSA_WITH_3DES_EDE_CBC_SHA"

Ignoring disabled cipher suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA for TLSv1.2
No available cipher suite for TLSv1.2
main, handling exception: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

So I think that the server want to use a cipher that the JDK won't allow.

This Jira issue seems similar https://bugs.openjdk.java.net/browse/JDK-8213846 -- the server drops the connection after the handshake is sent.

So the question now becomes: how to enable that cipher suite, which I have not figured out how to do!