CODEWITHSUNDEEP
apache-kafka
jagadish
jagadish

Reputation: 35

get the Message count and last offset of each partition of topic in Java

Using Admin Client to communicate with Kafka in Java code. we are able to do all the CRUD operations on particular topic. Our new requirement is to get the number of messages for topic, Size of all messages for Topic, Last Offset for each partition of Topic.

Input: Broker List and Topic Expected Output like below for 5 partitions

{
  "total": 10,
  "partitions": [
    3,
    2,
    1,
    3,
    1
  ],
  "offsets": [
    10,
    9,
    9,
    11,
    12
  ]
}

Total is the size of message in bytes.

Tried using the Consumer group ID. Here we will be able to get the data only if the topic is consumed.We want to get the topic details regardless it is consumed or Not.

Please suggest possible apporaches

Upvotes: 0

Views: 3645

Answers (1)

Nishu Tayal
Nishu Tayal

Reputation: 20830

Using Commandline, you can collect the offsets for each partition of a topic in following way:

kafka%~ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic1

It will return the offsets per partition as below:

mytopic1:2:828012
mytopic1:1:827041
mytopic1:3:829577
mytopic1:0:829184

Here you can refer more detail : https://cwiki.apache.org/confluence/display/KAFKA/System+Tools

Upvotes: 4

Related Questions