CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
TECH007
TECH007

Reputation: 179

Retry logic in kafka consumer

I have a use case where i consume certain logs from a queue and hit some third party API's with some info from that log , in case the third party system is not responding properly i wish to implement a retry logic for that particular log .

I can add a time field and repush the message to the same queue and this message will get again consumed if its time field is valid i.e less than current time and if not then get pushed again into queue.

But this logic will add same log again and again until retry time is correct and the queue will grow unnecessarily.

Is there is better way to implement retry logic in Kafka ?

Upvotes: 15

Views: 30621

Answers (3)

TECH007
TECH007

Reputation: 179

Yes, this could be one straight solution that I also thought of. But with this, we will end up in creating many topics as it is possible that message processing will fail again.

I solved this problem by mapping this use case to Rabbit MQ. In rabbit MQ we have the concept of retry exchange where if a message processing fails from an exchange then u can send it to a retry exchange with a TTL. Once TTL gets expired the message will move back to the main exchange and is ready to be processed again.

I can post some examples explaining how we can implement an exponential backoff message processing using Rabbit MQ.

Upvotes: 1

Pave
Pave

Reputation: 2428

You can create several retry topics and push failed task there. For instance you can create 3 topics with different delays in mins and rotate the single failed task till the max attempt limit reached.

‘retry_5m_topic’ — for retry in 5 minutes

‘retry_30m_topic’ — for retry in 30 minutes

‘retry_1h_topic’ — for retry in 1 hour

See more for details: https://blog.pragmatists.com/retrying-consumer-architecture-in-the-apache-kafka-939ac4cb851a

Upvotes: 11

ksv
ksv

Reputation: 67

In consumer, if it throws an exception, produce another message with attempt number 1. so next time when it is consumed, it has the property of attempt no 1. Handle it in the producer that, if it attempts more than your retry count, then stop producing it.

Upvotes: 1

Related Questions