CODEWITHSUNDEEP
azureazureservicebusazure-servicebus-queues
Sudsy
Sudsy

Reputation: 971

Fair Queuing of Publishers in Azure Service Bus

I am looking for a way to manage streams of messages from multiple publishers.

A contrived example is this.

Each of my publishers are submitting jobs that can be broken into smaller components that can be processed in any order. Each component takes several seconds to process.

Publisher A submits a job to a queue that consists of 1000 smaller components. Consumer X subscribes to the queue and starts processing the queue messages and sending the results to a result queue.

Half way through the job, Publisher B submits a job with 1000 components.

I would like Consumer X to now alternate between messages from Publisher A and Publisher B so that Publisher B does not need to wait for the first job to complete before receiving results and/ Publisher A just receives results more slowly now that Publisher B has a job. (Similar to fair queuing in networking)

This should cater for any number of Publishers.

If I then scale up the Consumers, each consumer should behave in the same way. They don't need to coordinate their actions to guarantee fair queuing, just a rough approximation is fine.

Is there a pattern for handling this in Service bus? Is there another azure technology I should be using instead?

Upvotes: 0

Views: 244

Answers (1)

LoekD
LoekD

Reputation: 11470

You could create a Topic to send the smaller component messages to.

Create multiple Subscriptions on that Topic. For example 2, and configure each to filter 50% of the messages. Subscription 1 would contain messages from Publisher A and Subscription 2 would contain messages from Publisher B. (here's how)

Then create a reader for each Subscription, resulting in the ability to process messages from multiple publishers at the same time, using multiple subscribers.

Upvotes: 1

Related Questions