CODEWITHSUNDEEP
javaapache-kafka
akd
akd

Reputation: 1627

What API on Kafka Client would allow one to reset the offset to the last committed one?

I am using java version of Kafka client versioned 0.10.0.0. In my case auto commit is turned off and we read using KafKaClient.poll(timout) api. We process all the messages and if everything goes well we call the commitSync() explicitly. But in the case of not able to process the messages we want to be able to re read from the last commit. Looking at the API I didn't find anything obvious. Can someone point me in the right direction here? Thanks.

Upvotes: 1

Views: 866

Answers (1)

Oliver Dain
Oliver Dain

Reputation: 9953

Use the seek method. Specifically, since you're auto-committing offsets you know which offset was last committed (or you could - just save that in a variable). If things are failing and you want to back up call seek passing the last committed offset.

You could also call the committed method to ask Kafka what the last saved offset was, but it's so easy to save it yourself you might as well.

It might be a better idea to defer the offset commit until you know things have worked. That's far safer as then it's fine if a machine crashes: if you always consume from the last committed offset you still can guarantee that each message will be seen at least once. If you commit before you know if you processed it you'll never be able to make that guarantee even with a manual seek; what if the machine crashed after you commit the offset but before you notice things didn't succeed.

Upvotes: 3

Related Questions