CODEWITHSUNDEEP
domain-driven-designmicroservicesevent-driven
reizals
reizals

Reputation: 35

How to handle publishing event when message broker is out?

I'm thinking how can I handle sending events when suddenly message broker go down. Please take a look at this code

using (var uow = uowProvider.Create())
{
    ...
    ...
    var policy = offer.Buy(customer);

    uow.Policies.Add(policy);

    // DB changes are saved here! but what would happen if...
    await uow.CommitChanges();

    // ...eventPublisher throw an exception?
    await eventPublisher.PublishMessage(PolicyCreated(policy));

    return true;        
}

IMHO if eventPublisher throw exception the event PolicyCreated won't be published. I don't know how to deal with this situation. The event must be published in system. I suppose that only good solution will be creating some kind of retry mechanism but I'm not sure...

Upvotes: 3

Views: 2000

Answers (3)

Eben Roux
Eben Roux

Reputation: 13256

I would like to elaborate a bit on the answers provided by both @Imran Arshad and @VoiceOfUnreason which are, of course, correct.

There are basically 3 patterns when it comes to publishing messages:

  • exactly once delivery (requires distributed transactions)
  • at most once delivery (no distributed transaction but may miss messages - like the actor model)
  • at least once delivery (no distributed transaction but may have duplicate messages)

The following is all in terms of your example.

For exactly once delivery both the database and the queue would need to provide the ability to enlist in distributed transactions. Some queues do not proivde this functionality out-of-the-box (like RabbitMQ) and even though it may be possible to roll your own it may not be the best option. Distributed transactions are typically quite slow.

For at most once delivery we have to accept that we may miss messages and I'm guessing that in most use-cases this is quite troublesome. You would get around this by tracking the progress and picking up the missed messages and resending them if required.

For at least once delivery we would need to ensure that the messages are idempotent. When we get a duplicate messages (usually quite an edge case) they should be ignored or their outcome should be the same as the initial message processed.

Now, there are a couple of ways around your issue. You could start a database transaction and make your database changes. Before you comit you perform the message sending. Should that fail then your transaction would be rolled back. That works fine for sending a single message but in your case some subscribers may have received a message. This complicates matters as all your subscribers need to receive the message or none of them get to receive it.

You could have your subscriber check whether the state is indeed true and whether it should continue processing. This places a burden on the subscriber and introduces some coupling. It could either postpone the action should the state not allow processing, or ignore it.

Another option is that instead of publishing the event you send yourself a command that indicates completion of the step. The command handler would perform the publishing and retry until all subscriber queues receive the message. This would require the relevant subscribers to ignore those messages that they had already processed (idempotence).

The outbox is a store-and-forward approach and will eventually send the message to all subscribers. You could have your outbox perhaps be included in the database transaction. In my Shuttle.Esb service bus one of the folks that used it came across a weird side-effect that I had not planned. He used a sql-based queue as an outbox and the queue connection was to the same database. It was therefore included in the database transasction and would roll back with all the other changes if not committed. Apologies for promoting my own product but I'm sure other service bus offerings may have the same functionality.

There are therefore quite a few things to consider and various techniques to mitigate the risk of a queue outage. I would, however, move the queue interaction to before the database commit.

Upvotes: 1

Imran Arshad
Imran Arshad

Reputation: 4002

For reliable system you need to save events locally. If your broker is down you have to retry and publish event.

There are many ways to achieve this but most common is outbox pattern. Just like your mail box your event/message stays locally and you keep retrying until it's sent and you mark the message published in your local DB.

you can read more about here Publish Events

Upvotes: 2

VoiceOfUnreason
VoiceOfUnreason

Reputation: 57279

You'll want to review Udi Dahan's discussion of Reliable Messaging without Distributed Transactions.

But very roughly, the PolicyCreated event becomes part of the unit of work; either because it is saved in the Policy representation itself, or because it is saved in an EventRepository that participates in the same transaction as the Policies repository.

Once you've captured the information in your database, retry the publish is relatively straight forward - read the events from the database, publish, optionally mark the events in the database as successfully published so that they can be cleaned up.

Upvotes: 0

Related Questions