CODEWITHSUNDEEP
apache-kafkaspring-kafkaoffset
srk
srk

Reputation: 5126

How to reprocess uncommitted records in Kafka

I am using Spring-kafka to interact with kafka. Consuming records from a topic: alerts from partition 0 from single consumer instance.

Here are my consumer properties related to offsets and commit mode

enable.auto.commit=false
auto.offset.reset=latest

And the ackmode is set to MANUAL_IMMEDIATE and syncCommits = true;// default value

Here is my visual of sample parition data p0

offset msg
0 msg1
1 msg2
2 msg3

While consuming messages sequentially, after the first message at offset 0 is read am not committing it for some business-use case reason, would like to reprocess it later. And, now current offset is at 1, msg processed successfuly so committing it. Now the offset is at 2 and it moves forward AFAIK. How do I reprocess the uncommitted message at offset 0 from the same topic. Is there an efficient way to re-process uncommitted record.

Upvotes: 2

Views: 949

Answers (1)

Gary Russell
Gary Russell

Reputation: 174554

You can perform a seek https://docs.spring.io/spring-kafka/docs/current/reference/html/#seek

However, you won't just reprocess record 0, you will get them all again.

You can send the record to another topic and consume it from there.

You can either do the send in your code, or the DefaultErrorHandler configured with a DeadLetterPublishingRecoverer can do it for you. Configure 0 retries and throwing an exception will cause it to be written to the dead letter topic immediately.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#annotation-error-handling

Also see non-blocking retries https://docs.spring.io/spring-kafka/docs/current/reference/html/#retry-topic

Upvotes: 2

Related Questions