CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
DaliMidou
DaliMidou

Reputation: 121

Duplicate offsets in a Kafka topic with more than one partition

I am using kafka_2.10-0.10.0.1 with zookeeper-3.4.10. I know that there are many types of offsets. I have two questions: - I want to know the type of the offset returned by ConsumerRecord.offset(). - If I use a topic created with 10 partitions, can I obtain a set of records with the same offset value? In my program, I need to obtain a list of records with different offset values. I want to know do I have to use a topic with a single partition to achieve this goal?

Upvotes: 0

Views: 2890

Answers (1)

Dmitry Minkovsky
Dmitry Minkovsky

Reputation: 38143

I want to know the type of the offset returned by ConsumerRecord.offset().

This is the offset of the record within the topic-partition the record came from.

If I use a topic created with 10 partitions, can I obtain a set of records with the same offset value?

Yes, you can seek to that offset in each partition and read the value. To do this, assign the topic-partitions you want to your consumer with Consumer#assign(), then use Consumer#seek() to see to the offset you want to read. When you poll(), the consumer will start reading from that offset.

I want to know do I have to use a topic with a single partition to achieve this goal?

You don't have to do this. You can read whatever offsets you want from whatever partitions you want.

Upvotes: 1

Related Questions