CODEWITHSUNDEEP
python-3.xpykafkastrimzi
Naggappan Ramukannan
Naggappan Ramukannan

Reputation: 2822

Getting import _rd_kafka error in all docker images while trying to connect to kafka via pykafka

HI i tried with multiple docker images like Ubuntu and python:3.8-alpine etc.. and everywhere I am getting an error as below while trying to connect to my kafka cluster (2.7) via pykafka library.

Environment info:

kafka server : 2.7 (installed via strimzi kafka in EKS)
kafka clinet: pykafka (2.8.0)
python version: 3.8, 3.7 in all version i get same error

Note: This error happens only when the code run's inside the container when I run outside i.e directly from my machine then it works fine

INFO:pykafka.topic:Could not load pykafka.rdkafka extension.
DEBUG:pykafka.topic:Traceback:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pykafka/topic.py", line 43, in <module>
    from . import rdkafka
  File "/usr/local/lib/python3.8/site-packages/pykafka/rdkafka/__init__.py", line 1, in <module>
    from .producer import RdKafkaProducer
  File "/usr/local/lib/python3.8/site-packages/pykafka/rdkafka/producer.py", line 7, in <module>
    from . import _rd_kafka
ImportError: cannot import name '_rd_kafka' from partially initialized module 'pykafka.rdkafka' (most likely due to a circular import) (/usr/local/lib/python3.8/site-pac
INFO:pykafka.cluster:Broker version is too old to use automatic API version discovery. Falling back to hardcoded versions list.
DEBUG:pykafka.cluster:Updating cluster, attempt 1/3
DEBUG:pykafka.connection:Connecting to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:Attempt 0: failed to connect to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:[Errno 2] No such file or directory
INFO:pykafka.connection:Retrying in 300ms.
INFO:pykafka.connection:Attempt 1: failed to connect to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:[Errno 2] No such file or directory
INFO:pykafka.connection:Retrying in 300ms.
INFO:pykafka.connection:Attempt 2: failed to connect to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:[Errno 2] No such file or directory
INFO:pykafka.connection:Retrying in 300ms.
WARNING:pykafka.broker:Failed to connect to broker at kafka-kafka-bootstrap.kafka:9093. Check the `listeners` property in server.config.

Upvotes: 0

Views: 333

Answers (1)

Naggappan Ramukannan
Naggappan Ramukannan

Reputation: 2822

This works with centos docker image, but not sure why did not work with other images like ubunut python3.8-alpine etc..

Upvotes: 0

Related Questions