CODEWITHSUNDEEP
apache-kafkakafka-consumer-apispring-kafka
Vipin Gupta
Vipin Gupta

Reputation: 223

How to handle kafka message if update message is received before create message is received?

We are having multiple partitions of kafka.

There are 2 services. 1st service creates data and sends a response back to UI and also sends create a message to Kafka. After a fraction of second, an update request is sent to 1st service and 1st service thus sends this update request to 2nd service. We don't define the partition on which data will be sent.

So let's say create message is sent to partition 1 and update message to sent to partition 2.

Now the issue is that sometimes update message is executed earlier than create message. As the create message was not executed and there was no data for it in the 2nd service, therefore, the update message fails.

Is there some way to ensure that the update message is executed after create message?

Upvotes: 2

Views: 1364

Answers (3)

Arpit Saxena
Arpit Saxena

Reputation: 56

Kafka guarantees ordering per partition and to send records to same partition they need to have the same key so that the DefaultPartitioner can put them on same partition.

In your case, I suspect that the key is not define, hence the records are going in round-robin fashion on any partition. You can override the default partitioner by overriding "partitioner.class" property of Producer.

From Kafka Definitive Guide -

When the key is null and the default partitioner is used, the record will be sent to one of the available partitions of the topic at random. A round-robin algorithm will be used to balance the messages among the partitions. If a key exists and the default partitioner is used, Kafka will hash the key (using its own hash algorithm, so hash values will not change when Java is upgraded), and use the result to map the message to a specific partition.

Upvotes: 1

nipuna
nipuna

Reputation: 4095

Kafka reads messages sequentially for same partition. So you need to publish Same context's messages to same partition. Use HashPartitioner and use same key for every steps of one context's messages. As an example, If you are creating, updating an Account, use AccountId as the key and use hash partitioner to produce.

One more thing, If you using compacted topics, you can not delete (tombstone) kafka messages by one null message, if all messages are not in one (same) partition.

Upvotes: 1

Gary Russell
Gary Russell

Reputation: 174544

No; you need to send them to the same partition; if you add a key to the records (e.g. customer id), the default partitioner will send them to the same partition.

Upvotes: 1

Related Questions