Senal Weerasinghe
Reputation: 131
LibrdKafka Producer unable to communicate with KafkaBroker(Java) in TLS communication
- I have created KeyStore for Java KafkaBroker and created required certificates for librdKafka and I'm getting below error when these components try to communicate with SSL/TLS.
ssl://172.25.93.195:11307/bootstrap: SSL handshake failed: s23_clnt.c:601: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol: (after 4ms in state CONNECT)
- Then I took a TCP dump and analyzed it. It gives below output. (librdkafka - 172.25.31.43, KafkaBroker- 172.25.31.42),
"1","0.000000","172.25.31.43","172.25.31.42","TCP","74","59116 → 26307 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=523149518 TSecr=0 WS=128"
"2","0.000007","172.25.31.42","172.25.31.43","TCP","74","26307 → 59116 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=523134399 TSecr=523149518 WS=128"
"3","0.000086","172.25.31.43","172.25.31.42","TCP","66","59116 → 26307 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=523149518 TSecr=523134399"
"4","0.000169","172.25.31.43","172.25.31.42","SSLv2","205","Client Hello"
"5","0.000171","172.25.31.42","172.25.31.43","TCP","66","26307 → 59116 [ACK] Seq=1 Ack=140 Win=15616 Len=0 TSval=523134399 TSecr=523149518"
"6","0.000762","172.25.31.42","172.25.31.43","TLSv1.2","73","Alert (Level: Fatal, Description: Unexpected Message)"
"7","0.000773","172.25.31.42","172.25.31.43","TCP","66","26307 → 59116 [FIN, ACK] Seq=8 Ack=140 Win=15616 Len=0 TSval=523134399 TSecr=523149518"
Consider about line number 4 and 6, it shows that librdkafka sending "Client Hello" via SSLv2 and KafkaBroker send ack with "TLSv1.2".
- To verify the generated keys, I tired to connect from openssl s_client, Its output was as below,
openssl s_client -debug -msg -connect 172.25.31.42:26307 -CAfile $SYSTEM_CONFIGS/MHV/ca-cert -key $SYSTEM_CONFIGS/MHV/producer_client.key -pass "pass:abc123" -cert $SYSTEM_CONFIGS/MHV/producer_client.pem
CONNECTED(00000003)
write to 0x696e10 [0x697d40] (139 bytes => 139 (0x8B))
0000 - 80 89 01 03 01 00 60 00-00 00 20 00 00 39 00 00 ......`... ..9..
0010 - 38 00 00 35 00 00 88 00-00 87 00 00 84 00 00 16 8..5............
0020 - 00 00 13 00 00 0a 07 00-c0 00 00 33 00 00 32 00 ...........3..2.
0030 - 00 2f 00 00 45 00 00 44-00 00 41 03 00 80 00 00 ./..E..D..A.....
0040 - 05 00 00 04 01 00 80 00-00 15 00 00 12 00 00 09 ................
0050 - 06 00 40 00 00 14 00 00-11 00 00 08 00 00 06 04 ..@.............
0060 - 00 80 00 00 03 02 00 80-00 00 ff f5 99 fc 8d ca ................
0070 - 6e e0 b8 23 d3 35 3d b9-9a 34 7a 12 93 14 b6 e5 n..#.5=..4z.....
0080 - 97 6d 0c f6 3a de c3 7a-88 bd d9 .m..:..z...
>>> SSL 2.0 [length 0089], CLIENT-HELLO
01 03 01 00 60 00 00 00 20 00 00 39 00 00 38 00
00 35 00 00 88 00 00 87 00 00 84 00 00 16 00 00
13 00 00 0a 07 00 c0 00 00 33 00 00 32 00 00 2f
00 00 45 00 00 44 00 00 41 03 00 80 00 00 05 00
00 04 01 00 80 00 00 15 00 00 12 00 00 09 06 00
40 00 00 14 00 00 11 00 00 08 00 00 06 04 00 80
00 00 03 02 00 80 00 00 ff f5 99 fc 8d ca 6e e0
b8 23 d3 35 3d b9 9a 34 7a 12 93 14 b6 e5 97 6d
0c f6 3a de c3 7a 88 bd d9
read from 0x696e10 [0x69d2a0] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 0a .......
28921:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:601:
SPEC
librdkafka version :- 1.1.0 OS version :- suse 11 SP2
- OpenSSL version:-
openssl
OpenSSL> version
OpenSSL 0.9.8j-fips 07 Jan 2009
OpenSSL>
- Linked libraries
linux-vdso.so.1 => (0x00007ffcf87a2000)
librdkafka.so.1 => /x01/devft/libs/librdkafka.so.1 (0x00007f81f77c7000)
librdkafka++.so.1 => /x01/devft/libs/librdkafka++.so.1 (0x00007f81f75a3000)
libavrocpp.so.1.8.3-SNAPSHOT.0 => /x01/devft/libs/libavrocpp.so.1.8.3-SNAPSHOT.0 (0x00007f81f72a3000)
librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x00007f81f707a000)
libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00007f81f6e62000)
libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x00007f81f6b06000)
libboost_date_time.so.1.59.0 => /x01/devft/libs/libboost_date_time.so.1.59.0 (0x00007f81f68f1000)
libboost_filesystem.so.1.59.0 => /x01/devft/libs/libboost_filesystem.so.1.59.0 (0x00007f81f66da000)
libboost_system.so.1.59.0 => /x01/devft/libs/libboost_system.so.1.59.0 (0x00007f81f64d6000)
libboost_serialization.so.1.59.0 => /x01/devft/libs/libboost_serialization.so.1.59.0 (0x00007f81f6283000)
libboost_thread.so.1.59.0 => /x01/devft/libs/libboost_thread.so.1.59.0 (0x00007f81f6060000)
libboost_chrono.so.1.59.0 => /x01/devft/libs/libboost_chrono.so.1.59.0 (0x00007f81f5e58000)
libclntsh.so.12.1 => /x01/app/oracle/product/12.1.0.2/client_1/lib/libclntsh.so.12.1 (0x00007f81f2e6e000)
libnnz12.so => /x01/app/oracle/product/12.1.0.2/client_1/lib/libnnz12.so (0x00007f81f2764000)
libgtest.so.0 => /x01/devft/libs/libgtest.so.0 (0x00007f81f24d5000)
libz.so.1 => /lib64/libz.so.1 (0x00007f81f22be000)
libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00007f81f2068000)
libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00007f81f1cc9000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f81f1ab0000)
librt.so.1 => /lib64/librt.so.1 (0x00007f81f18a7000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f81f16a3000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f81f1467000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f81f125e000)
libstdc++.so.6 => /x01/devft/libs/libstdc++.so.6 (0x00007f81f0f46000)
libm.so.6 => /lib64/libm.so.6 (0x00007f81f0ccc000)
libgcc_s.so.1 => /x01/devft/libs/libgcc_s.so.1 (0x00007f81f0ab5000)
libc.so.6 => /lib64/libc.so.6 (0x00007f81f073e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f81f7ac1000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f81f0520000)
libboost_filesystem.so.1.67.0 => /x01/devft/libs/libboost_filesystem.so.1.67.0 (0x00007f81f0307000)
libboost_system.so.1.67.0 => /x01/devft/libs/libboost_system.so.1.67.0 (0x00007f81f0103000)
libboost_program_options.so.1.67.0 => /x01/devft/libs/libboost_program_options.so.1.67.0 (0x00007f81efe98000)
libboost_iostreams.so.1.67.0 => /x01/devft/libs/libboost_iostreams.so.1.67.0 (0x00007f81efc7d000)
libnl.so.1 => /lib64/libnl.so.1 (0x00007f81efa2b000)
libmql1.so => /x01/app/oracle/product/12.1.0.2/client_1/lib/libmql1.so (0x00007f81ef7b4000)
libipc1.so => /x01/app/oracle/product/12.1.0.2/client_1/lib/libipc1.so (0x00007f81ef436000)
libons.so => /x01/app/oracle/product/12.1.0.2/client_1/lib/libons.so (0x00007f81ef1f0000)
libaio.so.1 => /lib64/libaio.so.1 (0x00007f81eefee000)
libclntshcore.so.12.1 => /x01/app/oracle/product/12.1.0.2/client_1/lib/libclntshcore.so.12.1 (0x00007f81eea76000)
Upvotes: 0
Views: 415
Answers (1)
Senal Weerasinghe
Reputation: 131
The root cause of this issue was that different versions or older versions of OpenSSL were installed in distributed machines.
E.g
|---------------------|------------------|------------------|
| Process | Machine | OpenSSL Version |
|---------------------|------------------|------------------|
| KafkaBroker | 168.25.33.12 | 1.0.1 |
|---------------------|------------------|------------------|
| librdKafka Producer | 168.25.33.13 | 0.9.8j |
|---------------------|------------------|------------------|
This can be resolved by upgrading openssl version to 1.0.1
Note :- To enable SSL support, minimum openssl version should be 1.0.1.
Upvotes: 1
Related Questions
- Kafka Producer Not Able to connect via Code
- kafka failed authentication due to: SSL handshake failed
- Kafka with SSL failed in producer
- Kafka SSL handshake failed in custom Java producer
- Connecting Kafka producer/consumer to broker via TLS
- librdkafka consumer and ssl configuration
- Kafka Producer in Java Error
- Kafka Producer does not appear to work properly
- KafkaProducer Connection refused
- Unable to communicate with kafka server using kafka Producer API