CODEWITHSUNDEEP
eventsmessagingevent-sourcingidempotent
Christian Paesante
Christian Paesante

Reputation: 504

Message idempotence - ordering considerations

Let's say we have a system were there's one producer that queues messages in a queue and multiple instances of the same consumer processing this events. Since we are in a Competing Consumers pattern we know that ordering is no longerd guaranteed. This means that we must ensure that our messages are idempotent.

From what I read here (under the Message ordering bullet point), we must ensure that message processing is idempotent.

Here's the questions:

An example: let's say that we have a "User Created" and a "User Deleted" messages (or any other couple of events which NEED to be processed in order). If we process "User deleted" before "User created" the user won't be deleted. Even if they're coming ordered in the event queue. Can really an idempotent processing/idempotent events give to a deleted user?

Another example. Let's suppose that we have an entity that have a score attribute. An user can modify the score. A second service consumes events of the "score entity" service and if the score reaches 100 the entity (or an entity reference) is inserted by the second service in the "Best category" entity. If the score reaches -20 the second service insert the score entity in the "Worse category". Having multiple instace of the second service can give an impredictable result if the "score 100" and "score -20" events are within a tiny interval of time. Any ideas on how to design the "score x" events or how to process these events?

Thank you so much for your help!

Upvotes: 4

Views: 1651

Answers (1)

Constantin Galbenu
Constantin Galbenu

Reputation: 17683

How can we design our message processing to be idempotent?

You should:

  • ignore messages that you have already "seen"; this means that the consumer should have a way of detecting that; it could, for example, keep a list of message IDs that it has processed (this means that every message should have an unique ID).
  • not throw an exception if a message does not change the state; for example, if you receive a second DeleteUser event (so the user is already deleted, a second delete should have no side effect) then you ignore it. Not every event can be idempotent, for example UpdateUserName should not be idempotent.

If we are saving every event in an event store, are there any consideration to be taken into account when designing each event's payload and the events aggregation to get the aggregate state?

You should design the events based on your Domain; the payload should not contain more information than it is needed by the domain. If additional information would make your readmodels a lot easier to implement then you could add it to the payload but be careful to mark it somehow as redundant.

An example: let's say that we have a "User Created" and a "User Deleted" messages (or any other couple of events which NEED to be processed in order). If we process "User deleted" before "User created" the user won't be deleted. Even if they're coming ordered in the event queue. Can really an idempotent processing/idempotent events give to a deleted user?

In this particular case, you can have an additional collection of deleted users; you can keep only their ID. When a CreateUser event arrives you can check to see if the user is already deleted by looking in the DeletedUsers collection and ignore it if the user is there. You can ignore every other event that comes for that user.

This solution is very dependent on the domain.

Another example. Let's suppose that we have an entity that have a score attribute. An user can modify the score. A second service consumes events of the "score entity" service and if the score reaches 100 the entity (or an entity reference) is inserted by the second service in the "Best category" entity. If the score reaches -20 the second service insert the score entity in the "Worse category". Having multiple instace of the second service can give an impredictable result if the "score 100" and "score -20" events are within a tiny interval of time. Any ideas on how to design the "score x" events or how to process these events?

This situation can be resolved by keeping in-the/attached-to readmodels (the two collections, best and worse) the timestamp/order/stream-version/whatever of the last processed event and ignore every event that is less-or-equal to that timestamp. In this way, if "score 100" is emitted after "score -20" but arrives first you should ignore the "score -20" because it has a lower timestamp, although it comes last.

This solution is generic but it relies on the existence of some ordering.

Upvotes: 4

Related Questions