Netty Socket SSLHandshakeException WRONG_VERSION_NUMBER

Question

I'm having issues with using a cert on a netty pipeline. The Netty pipeline is executed by Spring, but the SSL is only a part of the Netty portion.

for dependencies I'm using netty and boringssl

    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-all</artifactId>
        <scope>compile</scope>
        <version>4.1.70.Final</version>
    </dependency>
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-tcnative-boringssl-static</artifactId>
        <version>2.0.45.Final</version>
    </dependency>

I start my pipeline with adding the sslhandler

@Override
protected void initChannel(Channel ch) {
    if (sslProperties.isSecured())
        ch.pipeline().addLast("ssl-handler", getSslHandler(ch));
}

that gets the handler from a function

protected SslHandler getSslHandler(Channel ch) {
    try {
        final SslContext sslCtx = sslService.getSslContext();
        return sslCtx.newHandler(ch.alloc());
    } catch (IOException e) {
        log.debug("Failed to establish Ssl Context", e);
        ch.writeAndFlush("Failed to establish SSH Context");
        ch.close();
    }
    return null;
}

and finally makes the context

public SslContext getSslContext() throws SSLException {
    PrivateKey privateKey = getPrivateKey();
    X509Certificate[] certChain = getCertificateChain();
    try {
        return SslContextBuilder.forClient()
                .sslProvider(SslProvider.OPENSSL)
                .protocols("TLSv1", "TLSv1.1", "TLSv1.2", "TLSv1.3")
                .trustManager(InsecureTrustManagerFactory.INSTANCE)
                .keyManager(privateKey, certChain)
                .build();
    } catch (IOException e) {
        log.warn("Failed to establish Ssl Context");
        log.debug("Failed to establish Ssl Context", e);
        throw e;
    }
}

The PrivateKey and CertChain are constructed properly, but once a client tries to connect for the first time I get

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER

I've had this working in the past, connecting to a separate server, and it worked fine then. The Channel is Initializing and it crashes when trying to read from the socket. It does not hit my try catches, and only reports the below stacktrace.

I know that currently i'm dealing with a TLSv1.3 where before I don't know exactly what was used. In addition, I was using JDK 1.8 when this problem was discovered, I'm currently on JDK 17. I've noticed some people saying that not all versions of 1.8 support TLSv1.3.

2021-11-10 08:41:47,218 ERROR [nioEventLoopGroup-2-1] com.test.router.nio.pipeline.handler.InboundClientCompleteHandler: Exception Caught from inbound-client-complete-handler
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.traffic.GlobalChannelTrafficShapingHandler.channelRead(GlobalChannelTrafficShapingHandler.java:573)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.needWrapAgain(ReferenceCountedOpenSslEngine.java:1334)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1351)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1296)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1383)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1426)
    at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342)
    at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
    ... 25 common frames omitted
2021-11-10 08:41:47,218 DEBUG [nioEventLoopGroup-2-1] com.test.router.pipeline.handler.decoder.EventDecoder: Caught Exception in EventDecoder: 
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.traffic.GlobalChannelTrafficShapingHandler.channelRead(GlobalChannelTrafficShapingHandler.java:573)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.needWrapAgain(ReferenceCountedOpenSslEngine.java:1334)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1351)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1296)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1383)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1426)
    at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342)
    at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
    ... 25 common frames omitted

Answer 1

Does the server you are connecting to use Server Name Indication (SNI)? You'll sometimes get a javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER and a similar stacktrace if you don't enable SNI on the client.

To enable SNI you'd need to change:

protected SslHandler getSslHandler(Channel ch) {
    try {
        final SslContext sslCtx = sslService.getSslContext();
        return sslCtx.newHandler(ch.alloc());
    } catch (IOException e) {
        log.debug("Failed to establish Ssl Context", e);
        ch.writeAndFlush("Failed to establish SSH Context");
        ch.close();
    }
    return null;
}

To

protected SslHandler getSslHandler(Channel ch, String host, int port) {
    try {
        final SslContext sslCtx = sslService.getSslContext();
        return sslCtx.newHandler(ch.alloc(), host, port);
    } catch (IOException e) {
        log.debug("Failed to establish Ssl Context", e);
        ch.writeAndFlush("Failed to establish SSH Context");
        ch.close();
    }
    return null;
}