Event Sourcing and dealing with data dependencies

Question

Given a REST API with the following operations resulting in events posted to Kafka:

• AddCategory
• UpdateCategory
• RemoveCategory
• AddItem (refers to a category by some identifier)
• UpdateItem
• RemoveItem

And an environment where multiple users may use the REST API at the same time, and the consumers must all get the same events. The consumers may be offline for an extended period of time (more than a day). New consumers may be added, and others removed.

The problems:

• Event ordering (only workaround single topic/partition?)
  1. AddItem before AddCategory, invalid category reference.
  2. UpdateItem before AddCategory, used to be a valid reference, now invalid.
  3. RemoveCategory before AddItem, category reference invalid.
  4. ....infinite list of other concurrency issues.
• Event Store snapshots for fast resync of restarted consumers
  1. Should there be a compacted log topic for both categories and items, each entity keyed by its identifier?
  2. Can the whole compacted log topic be somehow identified as an offset?
  3. Should there only be one one entry in the compacted log topic, and the data of it contain a serialized blob of all categories and items given an offset (would require single topic/partition).
  4. How to deal with the handover from replaying the rendered entities event store to the "live stream" of commands/events? Encode offset in each item in the compacted log view, and pass that to replay from the live event log?

Are there other systems that fit this problem better?

Answer 1

I will give you a partial answer based on my experience in Event sourcing.

Event ordering (only workaround single topic/partition?)

1. AddItem before AddCategory, invalid category reference.

2. UpdateItem before AddCategory, used to be a valid reference, now invalid.

3. RemoveCategory before AddItem, category reference invalid.
4. ....infinite list of other concurrency issues.

All scalable Event stores that I know of guaranty events ordering inside a partition only. In DDD terms, the Event store ensure that the Aggregate is rehydrated correctly by replaying the events in the order they were generated. The Apache-kafka topic seems to be a good choice for that. While this is sufficient for the Write side of an application, it is harder for the Read side to use it. Harder but not impossible.

Given that the events are already validated by the Write side (because they represent facts that already happened) we can be sure that any inconsistency that appears in the system is due to the wrong ordering of events. Also, given that the Read side is eventually consistent with the Write side, the missing events will eventually reach our Read models.

So, first thing, in your case AddItem before AddCategory, invalid category reference, should be in fact ItemAdded before CategoryAdded (terms are in the past).

Second, when ItemAdded arrives, you try to load the Category by ID and if it fails (because of the delayed CategoryAdded event) then you can create a NotYetAvailableCategory having the ID equal to the referenced ID in the ItemAdded event and a title of "Not Yet Available Please Wait a few miliseconds". Then, when the CategoryAdded event arrives, you just update all the Items that reference that category ID. So, the main idea is that you create temporary entities that will be finalized when their events eventually arrive.

In the case of CategoryRemoved before ItemAdded, category reference invalid, when the ItemAdded event arrives, you could check that the category was deleted (by havind a ListOfCategoriesThatWereDeleted read model) and then take the appropriate actions in your Item entity - what depends on you business.