PatPanda
PatPanda

Reputation: 5070

identify retried message with spring kafka producer

What I would like to achieve:

I would like to identify the retried message using Spring Kafka when producing the retried message.

Background:

I am a Spring Kafka producer.

I produce messages that I put in a kafka broker.

Unfortunately, the kafka cluster is flaky, but not to a point where it is completely down. Usually, couple of retries will send the message successfully to Kafka.

The different consumers of the topic have the requirement to identify a message that has been retried.

Meaning, when they get a message I managed to produce without problems, no need for anything else.

However, if the message has been produced, but went thought the retry mechanism, I would like to identify / tag those a such.

Here is what I tried:

retries (defaults to Integer.MAX_VALUE): the maximum number of attempts to publish the message
delivery.timeout.ms (defaults to 120,000): the maximum time to wait for a message to be acknowledged before considering it failed
retry.backoff.ms (defaults to 100): the time to wait before retrying
retry.backoff.max.ms (defaults to 1,000): the maximum delay between consecutive retries

With this from the official doc, I went to use this construct:

 Properties props = new Properties();
    // other properties
    props.put(RETRIES_CONFIG, 20);
    props.put(RETRY_BACKOFF_MS_CONFIG, "500");
    props.put(DELIVERY_TIMEOUT_MS_CONFIG, "5000");

[...]
props.put(BOOTSTRAP_SERVERS_CONFIG, KAFKA_CONTAINER.getBootstrapServers());
    props.put(KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    props.put(VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    KafkaProducer<String, String> producer = new KafkaProducer<>(props);

While this would configure the retry, it does not distinguish a retried message.

Question:

How to distinguish, identify, tag that a message has been retried?

Upvotes: 0

Views: 33

Answers (0)

Related Questions