CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Divs
Divs

Reputation: 1618

kafka offset value differs between __consumer_offset and actual topic offset

Why does the offset value in actual topic is different than the offset value in __consumer_offset for the same topic? PFB the offset positions along with the commands used.

__consumer_offsets_13
=====================================
[root@node1 __consumer_offsets-13]# /opt/kafka_2.11-0.10.1.1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --print-data-log --files ./00000000000000000000.log | tail -n 5
_offset: 41368_ position: 4190035 CreateTime: 1532888732120 isvalid: true payloadsize: 28 magic: 1 compresscodec: NoCompressionCodec crc: 1789813648 keysize: 43 key: my_consumer_groupmy_topic payload: Ad�I��d�p-�

my_topic
=====================================
[root@node1 __consumer_offsets-13]# /opt/kafka_2.11-0.10.1.1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --print-data-log --files ../my_topic-0/00000000000000000000.log | tail -n 5
offset: 2080 position: 315620 CreateTime: 1532891670673 isvalid: true payloadsize: 118 magic: 1 compresscodec: NoCompressionCodec crc: 744326405 payload: {message_content_1}
_offset: 2081_ position: 315772 CreateTime: 1532891670673 isvalid: true payloadsize: 118 magic: 1 compresscodec: NoCompressionCodec crc: 2573656188 payload: {message_content_2}

What am I missing?

Upvotes: 0

Views: 572

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26950

The kafka.tools.DumpLogSegments tool prints the content of the logs.

The offset you see when running it against __consumer_offsets is the offset of the message that contains the consumer group offset. It is not the offset of the consumer group!

The actual offset of the consumer group is contained in the payload of that message. In your output, it displays as payload: Ad�I��d�p-� because it needs to be decoded.

See Kafka how to read from __consumer_offsets topic if you want to print the content of __consumer_offsets.

Upvotes: 1

Related Questions