CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Alana Storm
Alana Storm

Reputation: 166066

Kafka: How to Display Offsets

I'm super new to Kafka. I've installed kafka and zookeeper using homebrew on my mac, and I'm playing around with the quickstart guide.

I've been able to push messages onto Kafka using the following command and STDIN

kafka-console-producer --broker-list localhost:9092 --topic test

and I can read things off using

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

What's not clear to me is how I use offsets. It's my understanding that each message added to a topic will have a numerical, incremental offset value. However, if I try to do something like this 

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset 1

I get a non-zero status code and shows me no messages (other than the usual help/usage information)

I also can't get the latest or earliest keywords to work

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset earliest
kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset latest

Both of the above also return non-zero status codes.

Do I fundamentally misunderstand offsets? If not, is there a way to list all messages with their offsets? Finally -- what's the simpliest example of the --offset flag for the kafka-console-consumer?

Upvotes: 10

Views: 17417

Answers (6)

LIU YUE
LIU YUE

Reputation: 1987

here is the answer, just use '--property print.offset=true':

bin/kafka-console-consumer.sh --bootstrap-server XXXXXX --topic T-TEST --partition 0 --offset 0 --property print.offset=true

Upvotes: 1

yash arora
yash arora

Reputation: 31

You have to give partition value as well with offset in above command like this

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --partition 0 --offset 1

Upvotes: 1

yuranos
yuranos

Reputation: 9685

Since GetOffsetShell only works with PLAINTEXT, it can be inconvenient for many.
Good news is that additional properties, including print.offset seems to have made their way into 2.7 based on the communication in 9099 PR.
It must be possible to use print.offset=true now!

Upvotes: 5

p13rr0m
p13rr0m

Reputation: 1297

A nice command line tool that can display the offset with each message is kafkacat.

kafkacat -b localhost:9092 -C -t test -f 'Topic %t [%p] at offset %o: key %k: %s\n'

It will print out something like

Topic test [5] at offset 111: key "0171bf8102007900e33": {"Message": "1"} 
Topic test [2] at offset 123: key "070021b0f001f614c1b": {"Message": "2"}

Upvotes: 5

deed singh
deed singh

Reputation: 21

Partition is required when using offset

.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --partition 0 --offset 0 --property print.key=true --property print.timestamp=true

Upvotes: 2

OneCricketeer
OneCricketeer

Reputation: 191710

If you looked at the output after you gave the offset value, it would have said that you needed to specify a partition (at the top of the help section)

Topics are subdivided into partitions, and an offset of 1 could only exist on one partition out of potentially hundreds, so you must specify it

Regarding displaying the offsets, lookup the GetOffsetShell command syntax

Upvotes: 5

Related Questions