Facing error while Enabling SSL in SOLR 9.6.1 with Zookeeper 3.8.1

Question

I'm using SOLR with zookeeper in Ubuntu 24.04.1 server, installed SOLR with 1 shard, 2 replicas and 3nodes of zookeeper(localhost) and all are running in same server with basic solr authentication. Now, Im trying to enable SSL in SOLR and Zookeeper. But, after making the SSL related changes in SOLR and Zookeeper configrations, SOLR Admin UI not launching in browser. Also, getting some errors in log file(I've attached below)

I've updated the configuration of SOLR and Zookeeper as below,

1. created keystore with root, intermediate and server certificate (i'm using the same keystore as a trusted store as well)
2. updated solr.in.sh

SOLR_SSL_ENABLED=true
# Uncomment to set SSL-related system properties
# Be sure to update the paths to the correct keystore for your environment
SOLR_SSL_KEY_STORE=/opt/solr-9.6.1/server/etc/solr-ssl.jks
SOLR_SSL_KEY_STORE_PASSWORD=Password
SOLR_SSL_TRUST_STORE=/opt/solr-9.6.1/server/etc/solr-ssl.jks
SOLR_SSL_TRUST_STORE_PASSWORD=Password
# Require clients to authenticate
SOLR_SSL_NEED_CLIENT_AUTH=false
# Enable clients to authenticate (but not require)
SOLR_SSL_WANT_CLIENT_AUTH=false
# Verify client's hostname during SSL handshake
#SOLR_SSL_CLIENT_HOSTNAME_VERIFICATION=false
# SSL Certificates contain host/ip "peer name" information that is validated by default. Setting
# this to false can be useful to disable these checks when re-using a certificate on many hosts.
# This will also be used for the default value of whether SNI Host checking should be enabled.
SOLR_SSL_CHECK_PEER_NAME=true
# Override Key/Trust Store types if necessary
#SOLR_SSL_KEY_STORE_TYPE=PKCS12
#SOLR_SSL_TRUST_STORE_TYPE=PKCS12
#SOLR_SSL_RELOAD_ENABLED=true

3. jetty-ssl.xml

<Configure id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory$Server">
  <Call class="org.apache.solr.util.configuration.SSLConfigurationsFactory" name="current">
    <Call name="init" />
  </Call>
  <Call class="org.apache.solr.util.configuration.SSLConfigurationsFactory" name="current">
    <Get name="keyStorePassword" id="keyStorePassword"/>
    <Get name="trustStorePassword" id="trustStorePassword"/>
  </Call>
  <Set name="KeyStorePath">/opt/solr-9.6.1/server/etc/solr-ssl.jks<Property name="solr.jetty.keystore"/></Set>
  <Set name="KeyStorePassword">Password<Ref refid="keyStorePassword"/></Set>
  <Set name="TrustStorePath">/opt/solr-9.6.1/server/etc/solr-ssl.jks<Property name="solr.jetty.truststore"/></Set>
  <Set name="TrustStorePassword">Password<Ref refid="trustStorePassword"/></Set>
  <Set name="NeedClientAuth">false<Property name="solr.jetty.ssl.needClientAuth"/></Set>
  <Set name="WantClientAuth">false<Property name="solr.jetty.ssl.wantClientAuth"/></Set>
  <Set name="KeyStoreType">PKCS12<Property name="solr.jetty.keystore.type"/></Set>
  <Set name="TrustStoreType">PKCS12<Property name="solr.jetty.truststore.type"/></Set>
  <Set name="EndpointIdentificationAlgorithm"><Property name="solr.jetty.ssl.verifyClientHostName"/></Set>
</Configure>

4. jetty-ssl-context-reload.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_10_0.dtd">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <Call name="addBean">
<!--
 <Arg>
            <New id="keyStoreScanner" class="org.eclipse.jetty.util.ssl.KeyStoreScanner">
                <Arg><Ref refid="sslContextFactory"/></Arg>
                <Set name="scanInterval"><Property name="solr.jetty.sslContext.reload.scanInterval" default="30"/></Set>
            </New>
        </Arg>
-->
        <Arg>
           <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory$Server">
           <Set name="keyStorePath">/opt/solr-9.6.1/server/etc/solr-ssl.jks</Set>
           <Set name="keyStorePassword">Password</Set>
           <Set name="trustStorePath">/opt/solr-9.6.1/server/etc/solr-ssl.jks</Set>
           <Set name="trustStorePassword">Password</Set>
           <Set name="KeyStoreType">PKCS12</Set>
           <Set name="TrustStoreType">PKCS12</Set>
           </New>
        </Arg>
    </Call>
</Configure>

5. zoo.cfg (same for all 3 nodes of zookeeper except secureClientPort, dataDir)

secureClientPort=2181
dataDir=/opt/zookeeper/data/z1
dataLogDir=/opt/zookeeper/logs
tickTime=2000
initLimit=10
syncLimit=5

serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
maxClientCnxns=60

server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
4lw.commands.whitelist=mntr,conf,ruok

admin.enableServer=false
quormListenOnAllIPs=true

autopurge.snapRetainCount=3
autopurge.purgeInterval=1

ssl.client.enable=true
ssl.keyStore.location=opt/solr-9.6.1/server/etc/localhostjks/localkeystore.jks
ssl.keyStore.password=Password
ssl.keyStore.type=PKCS12
ssl.trustStore.location=opt/solr-9.6.1/server/etc/localhostjks/localkeystore.jks
ssl.trustStore.password=Password
ssl.trustStore.type=PKCS12
sslQuorum=true
quorum.cnxn.factory=org.apache.zookeeper.server.quorum.NettyServerCnxnFactory
clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
ssl.quorum.keyStore.location=opt/solr-9.6.1/server/etc/localhostjks/localkeystore.jks
ssl.quorum.keyStore.password=Password
ssl.quorum.keyStore.type=PKCS12
ssl.quorum.trustStore.location=opt/solr-9.6.1/server/etc/localhostjks/localkeystore.jks
ssl.quorum.trustStore.password=Password
ssl.quorum.trustStore.type=PKCS12
ssl.quorum.hostnameVerification=false
ssl.quorum.enabledProtocols=TLSv1.2

After making all the above changes, I've restarted SOLR, Zookeeper with below commands

1. zookeeper restart

/bin/bash /opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/data/z1/zoo.cfg
/bin/bash /opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/data/z2/zoo.cfg
/bin/bash /opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/data/z3/zoo.cfg

2. SOLR restart

/bin/bash /opt/solr/bin/solr restart -c -p 8983 -z localhost:2181,localhost:2182,localhost:2183 -m 5g -force

above commands are executed successfully but, SOLR UI not launching. I noticed below errors in Zookeeper log file and unable to fix this:

2024-10-14 15:54:16,182 [myid:] - INFO  [QuorumPeer[myid=3](plain=disabled)(secure=[0:0:0:0:0:0:0:0]:2183):o.a.z.s.q.FastLeaderElection@996] - Notification time out: 1600 ms
2024-10-14 15:54:16,277 [myid:] - WARN  [QuorumConnectionThread-[myid=3]-2:o.a.z.s.q.QuorumCnxManager@401] - Cannot open channel to 2 at election address localhost/127.0.0.1:3889
java.net.SocketException: Broken pipe
        at java.base/sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425)
        at java.base/sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445)
        at java.base/sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831)
        at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1035)
        at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeChangeCipherSpec(SSLSocketOutputRecord.java:231)
        at java.base/sun.security.ssl.OutputRecord.changeWriteCiphers(OutputRecord.java:187)
        at java.base/sun.security.ssl.ChangeCipherSpec$T10ChangeCipherSpecProducer.produce(ChangeCipherSpec.java:118)
        at java.base/sun.security.ssl.Finished$T12FinishedProducer.onProduceFinished(Finished.java:393)
        at java.base/sun.security.ssl.Finished$T12FinishedProducer.produce(Finished.java:377)
        at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440)
        at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:387)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-14 15:54:24,871 [myid:] - INFO  [ListenerHandler-localhost/127.0.0.1:3890:o.a.z.s.q.QuorumCnxManager$Listener$ListenerHandler@1076] - Received connection request from /127.0.0.1:46886
2024-10-14 15:54:25,046 [myid:] - INFO  [ListenerHandler-localhost/127.0.0.1:3890:o.a.z.s.q.UnifiedServerSocket$UnifiedSocket@266] - Accepted TLS connection from /127.0.0.1:46886 - NONE - SSL_NULL_WITH_NULL_NULL
2024-10-14 15:54:25,048 [myid:] - WARN  [ListenerHandler-localhost/127.0.0.1:3890:o.a.z.s.q.QuorumCnxManager@630] - Exception reading or writing challenge
java.net.SocketException: Socket is closed
        at java.base/sun.security.ssl.SSLSocketImpl.getInputStream(SSLSocketImpl.java:893)
        at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
        at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
        at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
        at java.base/java.io.DataInputStream.readLong(DataInputStream.java:410)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:602)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:555)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1085)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-14 15:54:33,271 [myid:] - WARN  [nioEventLoopGroup-4-1:i.n.u.c.DefaultPromise@593] - An exception was thrown by org.apache.zookeeper.server.NettyServerCnxnFactory$CertificateVerifier.operationComplete()
java.lang.NullPointerException: Cannot invoke "org.apache.zookeeper.server.ZooKeeperServer.serverStats()" because "this.this$0.zkServer" is null
        at org.apache.zookeeper.server.NettyServerCnxnFactory$CertificateVerifier.operationComplete(NettyServerCnxnFactory.java:467)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
        at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1883)
        at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1853)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1220)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-14 15:54:33,277 [myid:] - WARN  [nioEventLoopGroup-4-1:o.a.z.s.NettyServerCnxnFactory$CnxnChannelHandler@304] - Exception caught
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        ... 17 common frames omitted

Note: The same set of warnings are repeating multiple time in log file, Also I'm using Putty to connect this Linux server to install SOLR and Zookeeper and launching SOLR UI in windows machine. Before making above SSL, its works well.

Could anyone please help me to fix this issue, Also let me know if I missed to update other setting.

Thanks in Advance,