CODEWITHSUNDEEP
apache-kafkakafka-consumer-apispring-kafka
user1511956
user1511956

Reputation: 844

Retry with backoff time per consumed message that fails in Kafka

In Kafka, is it possible to set a backoff time per message, if processing of that message fails ? - So that I can process other messages, and try again later with the message that failed ? If I just put it back out in front of the topic, it will reappear very fast. I am using Kafka with Spring Boot.

Upvotes: 1

Views: 792

Answers (1)

Michael Heil
Michael Heil

Reputation: 18475

Kafka does not have any built-in capabilities for a backoff time when consuming data as far as I know.

As soon as you process other messages successfully and also commit them it will be difficult to re-read only those where the processing failed. Kafka topics are built to be consumed in sequence while guaranteeing the order of messages per TopicPartition.

What we usually do in such a scenario is to catch the Exception during the processing of the message and then send it into a seperate topic (together with a error code/hint) and continue the processing of later incoming messages. That way you can analyse the data later and, if necessary, move the messages from that other topic into your original topic again.

The insertion of the problematic messages from the seperate topic into your original input topic could be done through a simple batch job that you run from time to time or even using the command line tools provided by Kafka.

Upvotes: 1

Related Questions