CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
rayman
rayman

Reputation: 21616

Kafka retry on same topic

We would like to create a retry kafka mechanism for failures. I saw many introduced a way of have.multiple 'retry' topics. Was wondering why cant i simplify the flow by clone the message add into it a retry-counter field and just re-produce it on the same topic until reached X times and then exhausted.

What do I miss with that mechanism?

Upvotes: 2

Views: 3041

Answers (2)

Krzysztof Tomaszewski
Krzysztof Tomaszewski

Reputation: 1164

Basically you're missing a configurable delay that should be present in retrying strategy. The simple approach you have presented will lead to very high CPU usage during some outage (for example some service you depend on is unavailable for some minutes or hours). The best approach is to have exponential backoff strategy - with each retry you increase a delay after which a message processing is retried again. And this "delivery delay" is something Kafka doesn't support.

Upvotes: 1

UnP
UnP

Reputation: 108

Not Sure if I understand the question correctly. Nevertheless, I would suggest that you have some Kafka 'retry' strategies.

  • Messages in 'retry' topics are already sorted in ‘retry_timestamp’ order
  • Because postponing message processing in case of failure is not a trivial process
  • If you would like to postpone processing of some messages, you can republish them to separate topics, one for each with some delay value
  • The failed messages processing can be achieved by cloning the message and later republishing it to one of the retry topics
  • Consumers of retry topics could block the thread (unless it is time to process the message)

Upvotes: 0

Related Questions