CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Aliaxander
Aliaxander

Reputation: 2617

Usecases for manual offset management in Kafka

I'm trying to implement Kafka consumer on Java.

Assume that the consumer contains some message-processing logic that may throw an exception. In that case the consumer should sleep for some time and reprocess the last message.

My idea was to use manual offset management: an offset is not committed on fail, thus the consumer presumably will read from the old offset.

During testing I've found out that a message is actually read only once in spite of the fact that an offset is not committed. Last committed offset considered only on application restart.

My questions are:

Upvotes: 0

Views: 1396

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62310

KafkaConsumer keep the latest offsets in-memory, thus, if an exception occurs (and you recover from it) and you want to read a message a second time, you need to use seek() before polling a second time.

Committing offsets is "only" there, to preserve the offsets when the client is shut down or crashed (ie, offsets stored reliably vs. in-memory). On client start up, the latest committed offsets are fetched and than the client only used it's own in-memory offsets.

Manual offset management is useful, if you want to "bundle" offset commits with some other action (eg, a second "commit" in another system that must be in-sync with committed Kafka offsets).

Upvotes: 1

Related Questions