CODEWITHSUNDEEP
apache-kafkakafka-producer-apikafka-python
Moinuddin Quadri
Moinuddin Quadri

Reputation: 48047

Kafka: org.apache.zookeeper.KeeperException$NoNodeException while creating topic on multi server setup

I am trying to setup multi node Kafka-0.8.2.2 cluster with 1 Producer, 1 consumer and 3 brokers all on different machines.

While creating topic on producer, I am getting error as org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids. Complete console output is available here. There is no error in Kafka Producer's log.

Command I am using to run Kafka is:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic edwintest

Note: Zookeeper service is running on all the servers, and all three brokers are having Kafka servers running on them (Only Brokers need Kafka Server. Right?).

Configuration of my producer.properties is as:

metadata.broker.list=<IP.OF.BROKER.1>:9092,<IP.OF.BROKER.2>:9092,<IP.OF.BROKER.3>:9092

producer.type=sync

compression.codec=none

serializer.class=kafka.serializer.DefaultEncoder

Below are some of the many articles I was using as reference:

Upvotes: 1

Views: 5302

Answers (2)

Moinuddin Quadri
Moinuddin Quadri

Reputation: 48047

The issue was because I was trying to connect to the zookeeper of localhost. My understanding was zookeeper is needed to be running on producer, consumer, and the Kafka brokers, and the communication is done between producer -> broker and broker -> consumer via zookeeper. But that was incorrect. Actually:

Zookeeper and Kafka servers should be running only on on broker servers. While creating the topic or publishing the content to the topic, public DNS of any of the Kafka broker should be passed with --zookeeper option. There is no need to run Kafka server on producer or consumer instance.

Correct command will be:

./bin/kafka-topics.sh --create --zookeeper <Public-DNS>:<PORT> --replication-factor 1 --partitions 3 --topic edwintest

where: Public-DNS is the DNS of any of the Kafka broker and PORT is the port of the zookeeper service.

Upvotes: 0

TobiSH
TobiSH

Reputation: 2921

At the very first glance it seems like you're calling create topic to a local zookeeper which is not aware of any of your kafka-brookers. You should call ./bin/kafka-topics.sh --create --zookeeper <IP.OF.BROKER.1>:2181

Upvotes: 1

Related Questions