CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Avishek Bhattacharya
Avishek Bhattacharya

Reputation: 6974

Kafka Consumer writing to the same queue from where it is reading

I have a use case where I want to consume from a kafka topic and depending on some logic if I am not able to process the message right now, I want to enqueue the message back to the same topic from where it had been read

Something like this

Topic1 ---> Consumer ---> Can't process now

^ |Re-enqueues________|

Is it possible ?

Upvotes: 0

Views: 500

Answers (1)

Michael Heil
Michael Heil

Reputation: 18485

Yes, this is possible.

However, be aware that depending on your retention settings the re-ingested message might exist in the topic multiple times. Also, the consumer will consume all messages as long as it is running which could lead to the case that it has consumed all valid messages but keeps on re-ingesting the other messages over and over again.

The typical pattern to deal with messages that should be re-ingested into your pipeline is to send them to a dedicated Kafka topic. Once your consumer is fixed to be able to process those messages you can then have your consumer read that dedicated topic just once.

Upvotes: 2

Related Questions