failed to connect to my local kafka broker from a local docker container

Question

I have a question regarding connecting to kafka broker from docker.

I have zookeeper and kafka broker running on my local machine. I have a docker container running on the same local machine with --network=host I want to send message from inside the docker container to my local kafka broker.

From inside the docker, I can connect to zookeeper and seek the existing topicsby running: ./bin/kafka-topics.sh --zookeeper localhost:2181 --list

But I cannot connect to the kafka broker with either the kafka-console-consumer.sh or kafka-console-producer.sh,

by running : bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test I get:

[2019-10-17 19:12:04,097] WARN [Consumer clientId=consumer-1, groupId=console-consumer-99825] Error connecting to node aictjt:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: aictjt
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:259)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:326)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:439)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:105)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)

Any help?

Thanks

Answer 1

The reason for local docker container throwing "java.net.UnknownHostException" is, it is not able to connect to the kafka broker. This is connectivity issue.

It would work if you follow below steps,

• In kafka/config/server.properties, there is "listener" line which would be commented by default. i.e. listeners=PLAINTEXT://localhost:9092. uncomment this line.
• Later restart both kafka broker and zookeeper.

Note: In non local environment, if you are setting up stream for first time, and facing similar issue to connect to broker and observing UnknownHostException. Try restarting broker and zookeeper.

Answer 2

The above solutions didn't work for me. After spending some time i found the issue was because zookeeper and kafka instance was not running on my local. This ans is tested on mac but i think will work on all platform

If you installed your zookeeper and kafka via homebrew then:

First run zookeeper at your local by running command:

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

Then run

kafka-server-start /usr/local/etc/kafka/server.properties

After this the commands for kafka will run without the error

java.net.UnknownHostException: aictjt

For example

kafka-topics --list --bootstrap-server localhost:9092

For those who installed without homebrew:

Open terminal with root directory where you installed kafka

./bin/zookeeper-server-start.sh /usr/local/etc/kafka/zookeeper.properties
./bin/kafka-server-start.sh /usr/local/etc/kafka/server.properties

After this the commands for kafka will run without the error

java.net.UnknownHostException: aictjt

For example

./bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Answer 3

The solution I found was:

In kafka/config/server.properties

set listeners=PLAINTEXT://localhost:9092, note the default listeners=PLAINTEXT://:9092 worked for some cases but for this specific machine I got I have to put localhost there.

re-run the kafka broker with the updated config.

And the connection works from inside my docker container (with --network=host).