CODEWITHSUNDEEP
apache-kafka
Shivkumar Mallesappa
Shivkumar Mallesappa

Reputation: 3077

How to list all available Kafka brokers in a cluster?

I am writing a shell script to monitor kafka brokers.

I have gone through some links and found that if ZooKeeper contains a list of brokers, and if, in this list, the IP address is present, then a kafka broker is running.

I want a command that I can use in my shell script to get the broker list and check whether kafka is running.

Is there any curl command to get the kafka cluster status like elasticsearch?

Upvotes: 112

Views: 244628

Answers (11)

Udayendu
Udayendu

Reputation: 505

In KRaft based deployment, you can use the below command to check the status of a broker:

$ kafka-metadata-quorum --bootstrap-server <broker_name>:9092 describe --status

Repeat the with correct broker_name for all the nodes in the cluster.

Upvotes: 0

Jose Alban
Jose Alban

Reputation: 7926

On MacOS, can try:

brew tap let-us-go/zkcli

brew install zkcli

or

brew install let-us-go/zkcli/zkcli
brew link --overwrite zkcli

zkcli ls /brokers/ids
zkcli get /brokers/ids/1

Upvotes: 4

suryaanand nadendla
suryaanand nadendla

Reputation: 1

This window command will give you the list of the active brokers between brackets:

.\bin\windows\zookeeper-shell.bat localhost:2181  ls /brokers/ids

Upvotes: 0

Michael Heil
Michael Heil

Reputation: 18475

Using Confluent's REST Proxy API v3:

curl -X GET -H "Accept: application/vnd.api+json" localhost:8082/v3/clusters

where localhost:8082 is Kafka Proxy address.

Upvotes: 2

Iraj Hedayati
Iraj Hedayati

Reputation: 1677

If you are using new version of Kafka e.g. 2.3.3 (or Confluent 5.3.3 and higher), you can use

kafka-broker-api-versions  --bootstrap-server BROKER | grep 9092

You just need to pass one of the brokers.

For Confluent and Kafka interoperability see

https://docs.confluent.io/platform/current/installation/versions-interoperability.html

Upvotes: 14

jimijazz
jimijazz

Reputation: 2257

This command will give you the list of the active brokers between brackets:

./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids

Upvotes: 171

Aydin K.
Aydin K.

Reputation: 3367

echo dump | nc localhost 2181 | grep brokers

(replace localhost with the host where zookeeper is running)

Upvotes: 31

Chaffelson
Chaffelson

Reputation: 1269

Here are a couple of quick functions I use when bash scripting Kafka Data Load into Demo Environments. In this example I use HDP with no security, but it is easily modified to other environments and intended to be quick and functional rather than particularly robust.

The first retrieves the address of the first ZooKeeper node from the config:

ZKS1=$(cat /usr/hdp/current/zookeeper-client/conf/zoo.cfg | grep server.1)
[[ ${ZKS1} =~ server.1=(.*?):[0-9]*:[0-9]* ]]
export ZKADDR=${BASH_REMATCH[1]}:2181
echo "using ZooKeeper Server $ZKADDR"

The second retrieves the Broker IDs from ZooKeeper:

echo "Fetching list of Kafka Brokers"
export BROKERIDS=$(/usr/hdp/current/kafka-broker/bin/zookeeper-shell.sh ${ZKADDR} <<< 'ls /brokers/ids' | tail -1)
export BROKERIDS=${BROKERIDS//[!0-9 ]/}
echo "Found Kafka Broker IDS: $BROKERIDS"

The third parses ZooKeeper again to retrieve the list of Kafka Brokers Host:port ready for use in the command-line client:

unset BROKERS
for i in $BROKERIDS
do
DETAIL=$(/usr/hdp/current/kafka-broker/bin/zookeeper-shell.sh ${ZKADDR} <<< "get /brokers/ids/$i")
[[ $DETAIL =~ PLAINTEXT:\/\/(.*?)\"\] ]]
if [ -z ${BROKERS+x} ]; then BROKERS=${BASH_REMATCH[1]}; else 
BROKERS="${BROKERS},${BASH_REMATCH[1]}"; fi
done
echo "Found Brokerlist: $BROKERS"

Upvotes: 10

Olivier Refalo
Olivier Refalo

Reputation: 51435

I did it like this

#!/bin/bash

ZK_HOST="localhost"
ZK_PORT=2181


for i in `echo dump | nc $ZK_HOST $ZK_PORT | grep brokers`
do
    echo $i
    DETAIL=`zkCli -server "$ZK_HOST:$ZK_PORT" get $i 2>/dev/null | tail -n 1`
    echo $DETAIL
done

Upvotes: 0

Gangadhar
Gangadhar

Reputation: 33

To use zookeeper commands with shell script try

zookeeper/bin/zkCli.sh -server localhost:2181 <<< "ls /brokers/ids" | tail -n 1. The last line usually has the response details

Upvotes: 2

Panchu
Panchu

Reputation: 1259

Alternate way using Zk-Client:

If you do not prefer to pass arguments to ./zookeeper-shell.sh and want to see the broker details from Zookeeper CLI, you need to install standalone Zookeeper (As traditional Kafka do not comes up with Jline JAR).

Once you install(unzip) the standalone Zookeeper,then:

  • Run the Zookeeper CLI:
    $ zookeeper/bin/zkCli.sh -server localhost:2181 #Make sure your Broker is already running

  • If it is successful, you can see the Zk client running as:

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
  • From here you can explore the broker details using various commands:

$ ls /brokers/ids # Gives the list of active brokers
$ ls /brokers/topics #Gives the list of topics
$ get /brokers/ids/0 #Gives more detailed information of the broker id '0'

Upvotes: 86

Related Questions