CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-apiapache-zookeeper
Himanshu Sahu
Himanshu Sahu

Reputation: 366

How to find which consumer is assigned to which partition of a topic in kafka?

I am building a kafka manager tool and I need to check which topic-partition is assigned to which consumer in a consumer group.

Suppose there is consumer-Group group-A consuming topic topic-A with n partitions, so there can be multiple consumers in group-A hosted in different VM's. So how to find which partitioned is assigned to which consumer host? Is it possible in kafka 0.9.1?

Thanks in advance.

Upvotes: 9

Views: 7347

Answers (2)

Haim Raman
Haim Raman

Reputation: 12023

I know it does not directly answer the question, but still relevant.

If you want to see the IP address of the consumer that is currently connected to a partition use the following command

kafka-consumer-groups.sh --bootstrap-server my-kafka-1:9092 --group my-consumer-group  --describe --members

E.g.

# kafka-consumer-groups.sh --bootstrap-server my-kafka-1:9092 --group my-consumer-group  --members --describe --verbose | grep name-of-topic
my-consumer-group consumer-my-consumer-group-1-40adb273-fd3d-46a3-9cd4-9c8c4a4da486  /10.8.36.66     consumer-my-consumer-group-1  1               name-of-topic(1)
my-consumer-group consumer-my-consumer-group-1-47fc5618-7f07-4b3e-a2a9-cf3c26ee3536  /10.8.9.79      consumer-my-consumer-group-1  1               name-of-topic(2)
my-consumer-group consumer-my-consumer-group-1-005da938-7c59-44c3-8d4c-4371fa222d15  /10.8.36.22     consumer-pmy-consumer-group-1  1               name-of-topic(0)

Upvotes: 1

Shawn Guo
Shawn Guo

Reputation: 3228

You can check how $KAFKA_HOME/bin/kafka-consumer-groups.sh works and integrate its implementation into your kafka manager tool, this tool will show you detailed group owner information(for example, partition assignment, lag, IP).

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
page_visits_10k 0 500 3333 2833 consumer-1_/10.139.176.190
page_visits_10k 1 0 3334 3334 consumer-1_/10.139.176.190
page_visits_10k 2 0 3333 3333 consumer-1_/10.139.176.190

Upvotes: 8

Related Questions