CODEWITHSUNDEEP
javaspring-bootapache-kafkamicroservices
Zed
Zed

Reputation: 5921

Ensuring ordered processing of messages

So I have a situation with a microservice architecture where I need to guarantee that incoming messages that have common identifier would be processed in order they come from kafka:

      message2, message1 kafka
     ------------------------------
             |message1       |message2
             |               |
         Instace1         Instance2

In the example below, I have two instances of a service that are processing messages from kafka, but I want to ensure that message2 is only processed after message1.

Apparently, this situation is easily solved by configuring one instance to consume only from a particular partition which would store messages with the common indetifier:

message2, message1 kafka
--------------------------------
       | message2
       | message1
     Instance1        Instance2

Now the order is guaranteed, and message2 will never be processed before message1.

However, I was wondering if this issue could be solved another way, directly in code instead of relying on infrastructure? This looks like it could be a standard problem in microservice architecture but I'm not sure what would be the preferred approach to solve it ?

Upvotes: 1

Views: 511

Answers (3)

Babak Badnava
Babak Badnava

Reputation: 41

You can disable the auto-commit feature and commit the offset of the message you have used manually. Take a look at this link to see how to configure it. Then, by having a variable that holds the last used index of the message you can do what you want but you must be sure that one single instance of code has access to this variable at a time. You can use another microservice to store/protect this value using something like a semaphore.

So, each consumer waits until all of the messages previous to the current message will be consumed and then starts to consume the message to save the order of messages.

But this solution adds more complexity to the code and also what is the benefit of using more than one consumer in this case? In the best case, there is no difference, in case of performance, between using 1 consumer or 10 consumers if you want to save the order or messages because consumers must wait until previous messages arravial.

Upvotes: 0

orirab
orirab

Reputation: 3363

I'd suggest infrastructure as the more "correct" way to go, but solving this with code should be possible:

If you have a single producer of messages, attach to the message the identifier of the directly-preceding message and before consuming the message make sure you consumed the directly-preceding one before.

If you have multiple producers, this gets a bit more tricky, as you'd have to synchronise the identifiers.

Again, I suggest the infrastructure to be the more "correct" way of solving this (the less code you write, and the less complex, the less bugs you'll have).

Upvotes: 1

Mickael Maison
Mickael Maison

Reputation: 26950

Kafka only guarantees ordering within a partition.

So if you want "message1" to be processed before "message2", you need to ensure both messages end up on the same partition. Then any consumer reading these messages is guaranteed to see them in the order they were produced.

Upvotes: 1

Related Questions