CODEWITHSUNDEEP
c#queuemsmqsequenceservice-broker
Prathul
Prathul

Reputation: 88

Maintain order of message in outgoing queue - .Net

In my scenario, messages are coming in a predefined order. Once it enters our system, multiple receivers picks up the message from the incoming queue and processes it. Once the processing is done, the processed messages should be sent out in the same order as they had arrived. In a scaled up system, how do we ensure this?

The system requires high processing speed and throughput. In .net world, which queue would be ideal for this scenario?

Upvotes: 2

Views: 1415

Answers (1)

tom redfern
tom redfern

Reputation: 31800

It's a fundamental problem around ordered delivery - somewhere in your system you need to throttle everything down to a single thread. This is unavoidable.

Where you choose to do this can make a large difference to throughput. You could choose to make your entire message processor single-threaded. This would ensure order was maintained but at a cost of low throughput.

However, there is a way you can still process messages concurrently, but then you need to somehow assemble them in the correct order again. There is an integration design pattern called Resequencer - http://eaipatterns.com/Resequencer.html.

However the resequencer pattern this relies on you being able to stamp each message with a time-stamp or sequence number on the way into your system if there is nothing already in your messages to indicate ordering.

Additionally, is ordered delivery a requirement across the entire message set coming in from the queue? For example, it may be that only some of your messages have a need to be delivered in order.

Or it could be that you can group your messages into "sets" under a correlating identifier - within each set order needs to be maintained but you can still have concurrent processing on a "per-set" basis.

Upvotes: 4

Related Questions