CODEWITHSUNDEEP
apache-kafkaqueueapache-kafka-connectletterdead-letter
Durgesh O Mishra
Durgesh O Mishra

Reputation: 61

dead letter queue for kafka connect http sink

I am writing HTTP Sink plugin for Kafka connect. Its purpose is to send HTTP requests for each message in the configured Kafka topic. I want to send the message to dead letter queue in case HTTP request fails. Can I make use of dead letter queue configuration provided in sink plugin?

The reason for this question is that, it has been mentioned in Kafka connect documentation and several blogs that only errors in transformer and converter will be send to dead letter queue and not the ones during PUT. Since the task of sending the HTTP request is done in PUT. So I am thinking, is there a way to send failed HTTP messages to DLQ? If not, is it possible to send the message to some other Kafka topic for further processing?

Upvotes: 1

Views: 928

Answers (1)

Mahedi Hasan
Mahedi Hasan

Reputation: 11

According to @Neil this might be informative,

Kip 610 (implemented in Kafka 2.6) added DLQ support for issues when interacting with end system. Kip 298 added DLQ but only on issues prior to sink system interaction. Check the versions of your connect cluster and sink version and see if it supports it.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors

Upvotes: 1

Related Questions