Kafka Broker Failed authentication - SSL handshake failed

Question

Getting SSL errors in a cluster of three Kafka servers that communicate over SSL (only). Why is this happening / how can I fix it?

server.properties
listeners=SSL://some_host_name.corp.com:9092
inter.broker.listener.name=SSL
ssl.keystore.location=/some/path/to/keystore.p12
ssl.keystore.password=***
ssl.key.password=***
ssl.truststore.location=/some/path/to/server.truststore.jks
ssl.truststore.password=***
ssl.enabled.protocols=TLSv1.2
ssl.client.auth=required
ssl.keystore.type=PKCS12
ssl.truststore.type=JKS
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# Kafka is running with -Djavax.net.debug=ssl

# Assume localhost IP address is 1.2.3.4
18:31:29.479 [data-plane-kafka-socket-acceptor-ListenerName(SSL)-SSL-9092] DEBUG kafka.network.Acceptor - Accepted connection from /1.2.3.4:46732 on /1.2.3.4:9092 and assigned it to processor 1, sendBufferSize [actual|requested]: [212992|1048576] recvBufferSize [actual|requested]: [212992|1048576]
18:31:29.479 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG kafka.network.Processor - Processor 1 listening to new connection from /1.2.3.4:46732
Using SSLEngineImpl.
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal error: 80: problem unwrapping net record
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, SEND TLSv1.2 ALERT:  fatal, description = internal_error
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, WRITE: TLSv1.2 Alert, length = 2
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeInbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.4:9092-1.2.3.4:46732-108 key=sun.nio.ch.SelectionKeyImpl@33a4cffe] SSLEngine.closeInBound() raised an exception.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[na:1.8.0_181]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[na:1.8.0_181]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[na:1.8.0_181]
        at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561) ~[na:1.8.0_181]
        at org.apache.kafka.common.network.SslTransportLayer.handshakeFailure(SslTransportLayer.java:871) [kafka-clients-2.7.0.jar:na]
        at org.apache.kafka.common.network.SslTransportLayer.maybeProcessHandshakeFailure(SslTransportLayer.java:909) [kafka-clients-2.7.0.jar:na]
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:295) [kafka-clients-2.7.0.jar:na]
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) [kafka-clients-2.7.0.jar:na]
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) [kafka-clients-2.7.0.jar:na]
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [kafka-clients-2.7.0.jar:na]
        at kafka.network.Processor.poll(SocketServer.scala:923) [kafka_2.12-2.7.0.jar:na]
        at kafka.network.Processor.run(SocketServer.scala:826) [kafka_2.12-2.7.0.jar:na]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] INFO  o.a.kafka.common.network.Selector - [SocketServer brokerId=1] Failed authentication with /1.2.3.4 (SSL handshake failed)
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()

Elsewhere in the log of another cluster member, authentication succeeds. So why does it work between different nodes, but not from one node to itself?

20:07:39.479 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.6:9092-1.2.3.5:41182-0 key=sun.nio.ch.SelectionKeyImpl@69864b7d] SSL handshake completed successfully with peerHost '1.2.3.5' peerPort 41182 peerPrincipal '...' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384'
20:07:39.480 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.kafka.common.network.Selector - [SocketServer brokerId=3] Successfully authenticated with /1.2.3.5

Answer 1

I have solved the issue by moving the SSL listener away from the default port of 9092.

This leaves me with a different question: if I have configured only one listener, with SSL, on port 9092, why is there any PLAINTEXT activity on that same port?

The errors shown above were caused by each broker trying to connect to itself on port 9092, probably using PLAINTEXT, even though I had configured only SSL.

Also: what is the difference between settings (1) and (2) below?

# (1) The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
listeners=SSL://$auto_host:9092

# (2) The port the socket server listens on
port=9092