CODEWITHSUNDEEP
pythonpython-3.xrabbitmqqueuepika
Dima Voronenkov
Dima Voronenkov

Reputation: 53

RabbitMQ and Python 3: How to consume message from different queues with priority

I have two types of tasks task1 and task2; also I have two types of consumers ConsumerA and ConsumerB.

ConsumerA can process only tasks type task1. ConsumerB can process both types of tasks: task1 and task2.

I want the ConsumerA to process only tasks type task1.

I want the ConsumerB to process tasks type task2 first and process tasks type task1 only when there is no tasks type2.

How to design this solution with RabbitMQ. I think about using two queues, but I don't know how to ensure that it processes tasks type2 only when there are no tasks type1.

I have read all StackOverflow without any success.

Maybe someone solved this task?

Upvotes: 1

Views: 1113

Answers (1)

Rafael Aguilar
Rafael Aguilar

Reputation: 3279

Hello @Dima Voronenkov

You may want to check Priorities Queues and also Round-robin dispatching from RabbitMQ documentation.

With both you could create one queue per worker, and distribute task1 to both queues and to process task2 with higher priority than task1.

This could be expanded further if this approach meets your requirements.

A graphic description of RoundRobin method:

Ex

Edit 1: To avoid task1 piling up for ConsumerB you can assign to the task1 a time to live so when they "expire" and send them to a "dead-letter-queue", that actually points to ConsumerA queue, so when ConsumerB is busy, those task1 messages will be redirected to the other queue.

Going beyond, you can also give "more tickets" to ConsumerA queue, at least in RR standard definition, I do not know how to do it on RabbitMQ, but it can be achieved creating more ConsumerA like queues, to group them in the same cluster of queues.

Approach 2

Using Topics(check link in the bottom) and priorities could also be a good solution, however I have not tested both on the same exercise, I can not see why it would not work, check on below image:

ex2

In this scenario, only one queue is needed with one exchange, and ConsumerA will only consume task1 messages while ConsumerB will consume both messages giving a higher priority to task2.

Topics: https://www.rabbitmq.com/tutorials/tutorial-five-python.html

Upvotes: 1

Related Questions