CODEWITHSUNDEEP
apache-kafkaspring-kafka
vegemite4me
vegemite4me

Reputation: 6856

How to determine a Kafka consumer's offset

We have an issue where it appears that a Kafka consumer is not receiving messages published to a topic. (I say appears as I have not yet got to the bottom of this, and I could be wrong.)

I am using Spring for Apache Kafka and my consumer is actually a method annotated with @KafkaListener.

This issue is intermittent and I am having trouble recreating it.

Is there a way for me to look at the Kafka broker's logs, or any other tool to help me find out the offset for my consumer? I want concrete evidence that my consumer is receiving the message or not.

Upvotes: 9

Views: 18237

Answers (4)

martinho
martinho

Reputation: 3379

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer-group1

Upvotes: 0

dovka
dovka

Reputation: 1061

Cluster Manager for Apache Kafka, previously known as Kafka Manager shows the consumer offsets as well as consumer lags:

consumer lag

However the detailed view also shows detailed offsets by partition: consumer partitions with detailed offset info

CMAK: Cluster Manager on github

It has many features allowing to manage and monitor multiple kafka clusters, we use it for many years and are quite happy.

It is under active development and supports latest version of kafka.

It is heavier than running the kafka command line, but docker makes it much easier to setup

Upvotes: 0

Aleja_Vigo
Aleja_Vigo

Reputation: 980

Besides using ./kafka-consumer-groups as described in the other answer which is correct and perfectly valid you can also use a Windows GUI application that shows you the same information about Consumer_groups and their offset/lag:

http://www.kafkatool.com/download.html

Upvotes: 1

Sönke Liebau
Sönke Liebau

Reputation: 1973

Take a look at the kafka-consumer-groups tool, which can be used to check offsets and lag of consumers (consumer has to be active at the time you run this command).

./kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --new-consumer --describe --group console-consumer-55936

GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
console-consumer-55936         test                           0          6               6               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           1          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           2          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           3          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           4          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           5          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           6          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           7          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           8          1               1               0               consumer-1_/192.168.0.83

This should allow you to track whether anything is actually being consumed or not.

Upvotes: 12

Related Questions