CODEWITHSUNDEEP
apache-kafka
Drathier
Drathier

Reputation: 14509

Can I make sure Kafka doesn't accept two copies of the same message?

I'm writing messages along with timestamps to kafka. If I retry, the timestamp might change, and the producer that's writing, but the message content and message id is the same. The message id is generated before the message gets here, and it's a uuid.

How can I make sure kafka doesn't accept the second copy, if it successfully wrote to the topic, but the ack got lost, so the service up the chain retries? The consumers must not ever see the duplicate message.

Upvotes: 0

Views: 1770

Answers (1)

ilooner
ilooner

Reputation: 2550

In general there are two cases when the same message can be sent to Kafka:

  1. During normal operation your application intentionally sends messages with the same uuid to Kafka and you want Kafka to do deduplication.
  2. While you are sending a message to Kafka your code or Kafka brokers fail and you want to make sure the message you try to send again isn't duplicated, and also isn't lost.

I assume you are interested in case 2.. The Kafka developer's call case 2. exactly-once delivery. The latest versions of Kafka support transactions in order to enable exactly-once delivery. A complete explanation of how Kafka does this along with a code snippet can be found in this article by Confluent (the Kafka company).

Upvotes: 2

Related Questions