Kafka offset management on Kafka topic as well as local database

Question

I want to manage offset in Kafka topic as well as the database so that if I want to reprocess in the queue after a certain point I can. How can I proceed on this? Thanks in advance.

Answer 1

Given a PartitionInfo you should be able to tell your consumer to seekToBeginning or seek to an offset in that partition.

A ConsumerRecord knows it's topic, partition and offset. You could record these facts in a database.

But the catch here is if your topics are partitioned. Your data then will be chronological for that category. So if you have two partitions and partition essentially by last name, the name changes for the first half of the alphabet will be sequential, and the second half will be sequential, but it's non obvious how to get a single chronological view of the name changes across the system.

However, if you recorded partition and offset for a particular change in the database, you could seek to that partition and offset and reprocess the stream from that point.

(This becomes irrelevant if you only have one partition, but it's something to think about when/if your topic or streaming architecture needs multiple partitions)

Stepping back from the actual question into theory, I'm not really sure why you would want to do this, as consumer groups will record your committed offset to Kafka itself, thus if your stream processing app crashes you'll be able to pick up from where you left off without worry. This message committing either happens automatically, if you set the enable.auto.commit property, or you can control this manually if you call commitSync() on the consumer. Or you're trying to use an immutable data store (Kafka) as one would a mutable store, but that's just a bit of pure speculation based on the fact that you're not really descriptive with why you want to do the thing you want to do.