CODEWITHSUNDEEP
azure-functionsazure-cosmosdbcqrsevent-sourcingmaterialized-views
Markus S.
Markus S.

Reputation: 2812

Aggregates with Event Sourcing and CQRS

In order to process large amounts of telemetry data and still be able to perform quick queries on the data, I'm adopting the Event Sourcing / CQRS patterns using Azure Functions and Azure Cosmos DB.

In my architecture, the inbound telemetry stream gets stored in a Cosmos DB Collection acting as event store.

To create materialized views of the raw telemetry data, I use another Azure Function with Cosmos DB Trigger which gets active on all new documents stored in my event store performing transformations on those documents.

This is quite easy working on a document per document basis. Where it get's tricky is, when I need to reference other documents in order to calculate my materialized view.

For example, when the received telemetry events contain relative counter values (e.g. energy used in a particular operation). In my materialized view I want to have a document containing a total sum of all energy consumption.

Now an easy implementation would be to look at the current state of this document in my materialized view and just increment this value by the newly received value.

The problem which i could get using this approach is when i have to recalculate my materialized views because in a future version i need to generate some additional views.

For recalculation, I would simply touch all related documents I want to recalculate in my event store, triggering the Azure Function which calculates the materialized views again. This would result in documents entering this Azure Function which were processed before.

When recalculation occurs, my counter would now not be accurate anymore if I simply increment my sum as documents that are already part of the sum would get added again.

Ways to solve this recalculation-scenario (i thought of) would be:

Could you give me some advice on how to properly solve that kind of scenarios?

Upvotes: 2

Views: 1193

Answers (1)

Markus S.
Markus S.

Reputation: 2812

So, to summarize the comments by @Mikhail and @RomanEremin and my thoughts, this would be the way to handle those scenarios:

In case of recalculating views:

  • Delete existing aggregates and build from scratch replaying the events in the event store.

In case of event bus delivering "at-least-once" (the way Azure functions with CosmosDb Trigger behave as a result of underlying ChangeFeedProcessor):

  • Version 1: Keep track of event ids (document ids) which are part of the aggregate in aggregation document and ignore events that are already part of the aggregate.
  • Version 2: Provide a sequential version (sequence number) of the source event and store the version the aggregate is based on in the aggregation document. When calculating the aggregate, check this sequence number against the sequence number of the event. If the event has a lower sequence number than the aggregation document: ignore, otherwise recalculate aggregate and update sequence number the aggregate is based on.

Upvotes: 2

Related Questions