CODEWITHSUNDEEP
apache-kafka
wandermonk
wandermonk

Reputation: 7386

How to find the consumer topic and the group id from the __consumer_offsets topic in kafka?

I am trying to parse the logs from the __consumer_offsets topic in kafka. The idea is to find the group id, topic and the consumer which is creating load in my kafka cluster.

The command I am executing is below

bin/kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server brokers --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --new-consumer --consumer.config consumer.conf

Now the output looks like this 

[dedupeconsumergroup,daas.dedupe.avrosyslog.incoming,4]::[OffsetMetadata[8646,NO_METADATA],CommitTime 1538115746766,ExpirationTime 1538202146766]
[dedupeconsumergroup,daas.dedupe.avrosyslog.incoming,6]::[OffsetMetadata[8639,NO_METADATA],CommitTime 1538115746766,ExpirationTime 1538202146766]

Can someone help me in understanding this log or point me to the documentation. Thanks in advance.

Upvotes: 0

Views: 473

Answers (1)

rocky
rocky

Reputation: 5004

[dedupeconsumergroup,daas.dedupe.avrosyslog.incoming,6]::[OffsetMetadata[8639,NO_METADATA],CommitTime 1538115746766,ExpirationTime 1538202146766]

it means that consumer group with name dedupeconsumergroup read/commited offset 8639 on partition 6 in topic daas.dedupe.avrosyslog.incoming

which particular consumer read particular offset is probably little more complicated to find as you would have to know which partition was assigned to which consumer at a given point in time - it can change over time due to rebalancing.

Upvotes: 1

Related Questions