CODEWITHSUNDEEP
python-3.xapache-kafkaopensslconfluent-platform
richardbnd19
richardbnd19

Reputation: 13

Can't establish SSL connection to Kafka after upgrading to python 3.7

Code I have that successfully connects to Kafka with an SSL connection in Python 3.6.7 fails when using Python 3.7.3, with error message SSL: WRONG_VERSION_NUMBER. I would not expect code working in Python 3.6 to fail when in Python 3.7. I would like to know how to resolve this error and connect to Kafka via SSL with Python 3.7.3.

I have tried several things to troubleshoot:

Reproducing this problem could be fairly involved. It requires running Kafka and Zookeeper alongside two different, comparable version of Python, and the complete set of SSL credentials that each of those require. Thankfully, Docker can take care of much of this for us. I have created a Github Repo that contains a minimal set of files required to reproduce the error using only Docker desktop:

https://github.com/r-archer37/python-kafka-mre

The exact steps to reproduce the error are in the README. The short version is that there are two docker-compose files where the only difference is the version of the Jupyter-provided python-based docker image. Each runs a simple script that installs pykafka and then attempts to connect to the Kafka container. The container with python 3.6 will successfully connect to Kafka (console output looks like DEBUG:pykafka.connection:Successfully connected to b'kafka':9092) and the container with python 3.7 will fail to connect to Kafka (console output looks like INFO:pykafka.connection:Attempt 0: failed to connect to kafka:9092 ... INFO:pykafka.connection:[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)).

Fixes, and suggestions of things to try, are both welcome!

Edit: The solution appears to be to use a kafka docker image by a different organization, not Confluent.

Upvotes: 1

Views: 2367

Answers (3)

Damir Ismakov
Damir Ismakov

Reputation: 21

If you’re still having that trouble, you can use this:

!allows usage of insecure ciphers!

ctx = ssl.SSLContext(<protocol>)
ctx.set_ciphers(‘ALL:@SECLEVEL=0’)

and pass ctx to KafkaConsumer/KafkaProducer constructor as ssl_context argument

Worked properly for me

Upvotes: 0

trey signer
trey signer

Reputation: 1

This can be solved by generating the certificate key with keytool -keyalg RSA

Upvotes: 0

Devin Bjelland
Devin Bjelland

Reputation: 71

This is quite weird. Based on my investigation, I suspect the python upgrade is bringing to light an issue with Kafka, you might want to file a bug report with them. I was able to reproduce it working with the python container 3.6 and the failure with the 3.7.

I captured wireshark traces of both. With 3.6, the client sends the Client Hello tls message, and the server responds with a valid Server Hello, completing the handshake. With 3.7, when the client sends the Client Hello message, the server responds with 0x00 repeated. 0x00 0x00 is not a valid TLS version, hence the WRONG_VERSION_NUMBER that openssl reports.

When trying to create a TLS connection to kafka using the openssl client from either container, the server is also responding to the client handshake with just a series of 0x00 bytes. Openssl client command I used: openssl s_client -connect kafka:9092 -cert mre.pem -CAfile mre.pem -key mre.pem -state -debug -tls1_2

Upvotes: 3

Related Questions