CODEWITHSUNDEEP
microservicescqrsevent-sourcingeventstoredb
gabrielgiussi
gabrielgiussi

Reputation: 9595

An event store could become a single point of failure?

Since a couple of days I've been trying to figure it out how to inform to the rest of the microservices that a new entity was created in a microservice A that store that entity in a MongoDB.

I want to:

At first a message broker like RabbitMQ seems to be a good tool for the job but then I see the problem of commit the new document in MongoDB and publish the message in the broker not being atomic.

Why event sourcing? by eventuate.io: enter image description here

One way of solving this issue implies make the schema of the documents a bit dirtier by adding a mark that says if the document have been published in the broker and having a scheduled background process that search unpublished documents in MongoDB and publishes those to the broker using confirmations, when the confirmation arrives the document will be marked as published (using at-least-once and idempotency semantics). This solutions is proposed in this and this answers.

Reading an Introduction to Microservices by Chris Richardson I ended up in this great presentation of Developing functional domain models with event sourcing where one of the slides asked:

How to atomically update the database and publish events and publish events without 2PC? (dual write problem).

The answer is simple (on the next slide)

Update the database and publish events

This is a different approach to this one that is based on CQRS a la Greg Young.

The domain repository is responsible for publishing the events, this would normally be inside a single transaction together with storing the events in the event store.

I think that delegate the responsabilities of storing and publishing the events to the event store is a good thing because avoids the need of 2PC or a background process.

However, in a certain way it's true that:

If you rely on the event store to publish the events you'd have a tight coupling to the storage mechanism.

But we could say the same if we adopt a message broker for intecommunicate the microservices.

The thing that worries me more is that the Event Store seems to become a Single Point of Failure.

If we look this example from eventuate.io enter image description here

we can see that if the event store is down, we can't create accounts or money transfers, losing one of the advantages of microservices. (although the system will continue responding querys).

So, it's correct to affirmate that the Event Store as used in the eventuate example is a Single Point of Failure?

Upvotes: 11

Views: 4398

Answers (5)

Anthony Anyanwu
Anthony Anyanwu

Reputation: 1047

Not particularly a mongodb solution but have you considered leveraging the Streams feature introduced in Redis 5 to implement a reliable event store. Take a look this intro here

I find that it has rich set of features like message tailing, message acknowledgement as well as the ability to extract unacknowledged messages easily. This surely helps to implement at least once messaging guarantees. It also support load balancing of messages using "consumer group" concept which can help with scaling the processing part.

Regarding your concern about being the single point of failure, as per the documentation, streams and consumer information can be replicated across nodes and persisted to disk (using regular Redis mechanisms I believe). This helps address the single point of failure issue. I'm currently considering using this for one of my microservices projects.

Upvotes: 0

Naveen Santhanavel
Naveen Santhanavel

Reputation: 439

How about if we have two event stores, and whenever a Domain Event is created, it is queued onto both of them. And the event handler on the query side, handles events popped from both the event stores.

Ofcourse every event should be idempotent. But wouldn’t this solve our problem of the event store being a single point of entry?

Upvotes: 1

ra1f
ra1f

Reputation: 91

You could also create a flag for each entry inside of the event store which tells if this event was already published. Another process could poll the event store for those unpublished events and put them into a message queue or topic. The disadvantage of this approach is that consumers of this queue or topic must be designed to de-duplicate incoming messages because this pattern does only guarantee at-least-once delivery. Another disadvantage could be latency because of the polling frequency. But since we have already entered the eventually consistent area here this might not be such a big concern.

Upvotes: 1

Udi Dahan
Udi Dahan

Reputation: 12067

We handle this with the Outbox approach in NServiceBus:

http://docs.particular.net/nservicebus/outbox/

This approach requires that the initial trigger for the whole operation came in as a message on the queue but works very well.

Upvotes: 2

Akira
Akira

Reputation: 4071

What you are facing is an instance of the Two General's Problem. Basically, you want to have two entities on a network agreeing on something but the network is not fail safe. Leslie Lamport proved that this is impossible.

So no matter how much you add new entities to your network, the message queue being one, you will never have 100% certainty that agreement will be reached. In fact, the opposite takes place: the more entities you add to your distributed system, the less you can be certain that an agreement will eventually be reached.

A practical answer to your case is that 2PC is not that bad if you consider adding even more complexity and single points of failures. If you absolutely do not want a single point of failure and wants to assume that the network is reliable (in other words, that the network itself cannot be a single point of failure), you can try a P2P algorithm such as DHT, but for two peers I bet it reduces to simple 2PC.

Upvotes: 6

Related Questions