CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
b15
b15

Reputation: 2351

Kafka consumer crashing after processing but before committing offset?

I'm doing some POC work with Kafka and am researching the exactly once features using the transactional API and am still stuck on a few questions. What happens if a consumer crashes after processing a message but before committing its offset? It seems like the next run would inevitably start from the wrong message and a duplicate message would get through. How do I deal with this scenario?

Upvotes: 1

Views: 996

Answers (1)

Manoj Vadehra
Manoj Vadehra

Reputation: 846

What happens if a consumer crashes after processing a message but before committing its offset?

Kafka: Definitive Guide mentions an option that if the message processing involves writing messages to a DB, we can write the processed offsets as well to the DB and use that offset in the recovery phase by seeking (seek()) to the offset from which we want to poll from Kafka.

That said, you can have a unique identifier in each message so that the consumer(s) can check if the message has been processed before or not. This (duplicate handling) should always be implemented in the consumer code.

Upvotes: 4

Related Questions