CODEWITHSUNDEEP
apache-kafkaevent-sourcing
M4V3N
M4V3N

Reputation: 629

How do I implement Event Sourcing using Kafka?

I would like to implement the event-sourcing pattern using kafka as an event store. I want to keep it as simple as possible.

The idea:

My app contains a list of customers. Customers an be created and deleted. Very simple. When a request to create a customer comes in, I am creating the event CUSTOMER_CREATED including the customer data and storing this in a kafka topic using a KafkaProducer. The same when a customer is deleted with the event CUSTOMER_DELETED.

Now when i want to list all customers, i have to replay all events that happened so far and then get the current state meaning a list of all customers.

I would create a temporary customer list, and then processing all the events one by one (create customer, create customer, delete customer, create customer etc). (Consuming these events with a KafkaConsumer). In the end I return the temporary list.

I want to keep it as simple as possible and it's just about giving me an understanding on how event-sourcing works in practice. Is this event-sourcing? And also: how do I create snapshots when implementing it this way?

Upvotes: 1

Views: 1558

Answers (2)

posthumecaver
posthumecaver

Reputation: 1863

Theoretically you can do Event Sourcing with Kafka as you mentioned, replaying all Events in the application start but as you mentioned, if you have 100 000 Events to reach a State it is not practical.

As it is mentioned in the previous answers, you can use Kafka Streaming KTable for sense of Event Sourcing but while KTable is hosted in Key/Value database RockDB, querying the data will be quite limited (You can ask what is the State of the Customer Id: 123456789 but you can't ask give me all Customers with State CUSTOMER_DELETED).

To achieve that flexibility, we need help from another pattern Command Query Responsibility Segregation (CQRS), personally I advice you to use Kafka reliable, extremely performant Broker and give the responsibility for Event Sourcing dedicated framework like Akka (which Kafka synergies naturally) with Apache Cassandra persistence and Akka Finite State Machine for the Command part and Akka Projection for the Query part.

If you want to see a sample how all these technology stacks plays together, I have a blog for it. I hope it can help you.

Upvotes: 0

OneCricketeer
OneCricketeer

Reputation: 192013

when i want to list all customers, i have to replay all events that happened so far

You actually don't, or at least not after your app starts fresh and is actively collecting / tombstoning the data. I encourage you to lookup the "Stream Table Duality", which basically states that your table is the current state of the world in your system, and a snapshot in time of all the streamed events thus far, which would be ((customers added + customers modified) - customers deleted).

The way you implement this in Kafka would be to use a compacted Kafka topic for your customers, which can be read into a Kafka Streams KTable, and persisted in memory or spill to disk (backed by RocksDB). The message key would be some UUID for the customer, or some other identifiable record that cannot change (e.g. not name, email, phone, etc. as all this can change)

With that, you can implement Interactive Queries on it to scan or lookup a certain customer's details.

Upvotes: 2

Related Questions