CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
sparrovv
sparrovv

Reputation: 7824

How to consume messages from a kafka topic from the specific point with console consumer?

I have a topic that has quite a big retention and loads of data stored. I'd like to read messages only from a specific point in time.

I know it's possible with java API, but I'd like to use kafka-console-consumer as I just want to check something quickly.

I could use --from-beginning flag but I won't to avoid loading loads of data that then I'd need to filter out.

Upvotes: 0

Views: 2115

Answers (1)

sparrovv
sparrovv

Reputation: 7824

The quickest way I've found is to create a consumer group, reset it to the time I'm interested in, and then use console-consumer using that new group, these are the commands:

# create a new group and reset the topic offset for the group to a specific date
kafka-consumer-groups --bootstrap-server localhost:9092 \
  --topic $topic \
  --reset-offsets \
  --group $randomname \
  --to-datetime 'YYYY-MM-DDTHH:mm:SS.sss' \
  --execute

and then:

# start consuming by providing the newly created group
kafka-console-consumer --bootstrap-server localhost:9092 \
  --group $randomname \
  --topic $topic

I'm using kafka-cluster 1.1, and tbh not sure if those flags are available for older versions.

Upvotes: 4

Related Questions