CODEWITHSUNDEEP
apache-kafka
Juan John Mathews
Juan John Mathews

Reputation: 736

Find broker id used in the Kafka cluster

I want to know the list of taken broker ids in a kafka cluster. For example, in a cluster with 10 nodes if I create a topic with 10 partitions(or more) I can see from the output of a describe topic command, the brokers to which it has been assigned. 

./bin/kafka-topics --describe --zookeeper <zkconnect>:2181 --topic rbtest3

Can I collect this information without creating a topic?

Upvotes: 6

Views: 24175

Answers (7)

FadingOffset
FadingOffset

Reputation: 151

Using the Zookeeper CLI

sh /bin/zkCli.sh -server zookeeper-1:2181 ls /brokers/ids

Then to get details of the broker, you can use the "get" with the ids that was the output generated from the previous command

sh /bin/zkCli.sh -server zookeeper-1:2181 get /brokers/ids/<broker-id>

Upvotes: 2

Marina
Marina

Reputation: 4064

If you want to know what is the broker ID of a specific broker - the easiest way is to look at its controller.log, I found:

cat /var/log/kafka/controller.log
[2021-02-18 13:20:22,639] INFO [ControllerEventThread controllerId=1003] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
[2021-02-18 13:20:22,646] DEBUG [Controller id=1003] Broker 1002 has been elected as the controller, so stopping the election process. (kafka.controller.KafkaController)

controllerId=1003 ---> this is your brokerID (1003)

[substitute your path to the kafka logs, of course ...]

Upvotes: 2

user10047195
user10047195

Reputation: 31

#kafka broker id
cat $KAFKA_HOME/logs/meta.properties

Upvotes: 3

NYCeyes
NYCeyes

Reputation: 5659

You also can use the zookeeper-shell.sh script that ships with the Kafka distribution, like this:

linux$ ./zookeeper-shell.sh zookeeper-IPaddress:2181 <<< "ls /brokers/ids"

Just add the IP address of any of your Zookeeper servers (and/or change the port if necessary, for example when running multiple Zookeeper instances on the same server).

This alternative can be useful when, for example, you find yourself inside a container (Docker, LXC, etc.) that is exclusively running a Kafka client; but Zookeeper itself is somewhere else (say, in a different container).

I hope it helps. =:)

Upvotes: 8

Luciano Afranllie
Luciano Afranllie

Reputation: 4253

You can get list of used broker ids using zookeeper cli.

zookeeper-3.4.8$ ./bin/zkCli.sh -server zookeeper-1:2181 ls /brokers/ids | tail -1
[0]

Upvotes: 9

medium
medium

Reputation: 4236

You can run the following command:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic=<your topic> --broker-list=<your broker list>  --time=-2

This will list all of the brokers with their id and the beginning offset.

Upvotes: 1

amethystic
amethystic

Reputation: 7079

you can use kafka-manager, an open-source tool powered by yahoo.

Upvotes: 1

Related Questions