Java 11 gives Unsupported handshake message: server_hello_done on HTTPS connection with client certificate

Question

We have a Java applet (running under tomcat) which makes calls to third parties. One of them uses a client certificate to authenticate. This was working under Java 8, but we recently upgraded the system to Java 11 and it doesn't any more. The error is

Unsupported handshake message: server_hello_done

(which is odd, since I thought server_hello_done was a valid part of the handshake)

We did have an issue with the java keystore after upgrade. The service failed saying it wasn't a valid PCKS12 stream. Listing the contents with keytool worked, but with warning

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /path/to/keystore -destkeystore /path/to/keystore -deststoretype pkcs12"

We used the suggested command and it now opens the keystore OK, but we get the handshake error.

The stack trace back to our code is:

Unsupported handshake message: server_hello_done
javax.net.ssl.SSLProtocolException: Unsupported handshake message: server_hello_done
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:126)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:446)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)

We are using the apache http client (org.apache.http.impl.client.CloseableHttpClient). The next line in the stack trace is simply a call to client.execute() in our code.

It also includes

Caused by: java.lang.UnsupportedOperationException: Not supported yet.
        at java.base/sun.security.ssl.HandshakeHash$CloneableHash.archived(HandshakeHash.java:616)
        at java.base/sun.security.ssl.HandshakeHash$T12HandshakeHash.archived(HandshakeHash.java:546)
        at java.base/sun.security.ssl.HandshakeHash.archived(HandshakeHash.java:188)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:581)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:740)
        at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
        at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)

The third party confirms they see "no shared cipher" in their logs, but say they support a broad range of ciphers ("SSL3, TLS 1.0, 1.1, or 1.2. Ciphers are marked as HIGH:MEDIUM:!aNULL:!eNULL:@STRENGTH"). I think we support most of that except SSL3. As far as I know we have default settings in java 11 for such. Tried activating SSLv3 temporarily but couldn't connect (although since trying that i can't connect from that machine at all (times out) even after reverting, so that may not say much - I tried from a test machine not the production one for that).

Any ideas? Am I on the right lines to keep looking at ciphers or is there something I'm missing?

Answer 1

It turned out that all that was needed was a restart of tomcat! But, since this was a production server doing many things I didn't want to do that as step one. But it fixed it.

My best guess as to what happened was:

• The keystore was in the wrong format for the updated java version
• The attempt to use the invalid keystore put tomcat/java into some weird state
• Updating the keystore allowed java to use it, but it was still in some weird state
• Restart fixed it.

What is really weird is that it behaved the same way even against a different IP (I tried against the third party's Test server) and with a different keystore (I made a copy of the updated keystore and did a request which used that). The initial error was "Stream is not a valid PKCS12 keystore" and the stack trace went back to the line that tried to open the keystore in our code. Having fixed the keystore format, it got further in our code (stack trace now went back to client.execute()) but it failed with _Unsupported handshake message: server_hello_done_.

I created a copy of the whole tomcat folder (including jre), with initially the original (invalid) keystore, changed it to run on another port and started it up next to this tomcat on the same machine. It behaved the same but then worked after a restart. This morning i restarted the main service and now it works