CODEWITHSUNDEEP
pythondockerapache-kafka
Hiba Rehman
Hiba Rehman

Reputation: 137

why kafka producer is showing me error kafka.conn:DNS lookup failed for <container id>:9092?

I expose the port 9092 then i run the kafka broker inside docker. But When I run the python script i get the errors 

ERROR:kafka.conn:DNS lookup failed for b5c5b06f6761:9092 (AddressFamily.AF_UNSPEC)

I tried docker ip and machine ip instead of localhost but gives same error.

Here is my code.

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],
                         value_serializer=lambda x:
                         dumps(x).encode('utf-8'))

producer.send('vtintel', value={'id':123})

Upvotes: 6

Views: 10116

Answers (4)

tnusraddinov
tnusraddinov

Reputation: 750

Same issue with bitnami/kafka. But then I realized that I need to enable accessing Kafka with external clients in docker-commpose.yml. For more info see https://hub.docker.com/r/bitnami/kafka/ 'Accessing Kafka with internal and external clients' part.

To do so, add the following environment variables to your docker-compose:

    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
+     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+     - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
+     - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

And expose the extra port:

    ports:
      - '9092:9092'
+     - '29092:29092'


and access it by 'localhost:29092' not 'localhost:9092' in your python-kafka code.

Incase it's not obvious, that should be added to the kafka container (not the zookeeper container)

Upvotes: 8

m umar Khan
m umar Khan

Reputation: 145

Though it's a late reply it might help anyone later. I have the same problem as you were facing I was using dockerized implementation of edX. So to fix that issue just add the following lines in
/etc/hosts
file of your docker container. first your IP Address then the thing for which the lookup is being failed. for example in your case lookup is being failed for b5c5b06f6761 so:

173.16.18.22     b5c5b06f6761

Note: Here I am using dummy IP address.

Upvotes: 2

OneCricketeer
OneCricketeer

Reputation: 191844

Docker only handles DNS within its own network, not from your host

You need to Kafka to advertise itself externally (on localhost), which is different than just a port forward

And as far as I can tell -p 9092:9092 is not a port even exposed by the container image you're using

Upvotes: 2

Sameer Killamsetty
Sameer Killamsetty

Reputation: 47

I had a similar problem earlier with latest kafka versions. Try mentioning the local address as '127.0.0.1' instead of 'localhost'. this might help.

Upvotes: -1

Related Questions