Why Doesn't Message Priority Sort My Messages In RabbitMQ?

Question

I run elixir program with producer and consumer. And I set priority for my messages. But priority is not working.

My program:

Publisher:

Basic.publish(channel, "my_exchange", "my_queue", "1", persistent: true, priority: 2)
Basic.publish(channel, "my_exchange", "my_queue", "2", persistent: true, priority: 3)
Basic.publish(channel, "my_exchange", "my_queue", "3", persistent: true, priority: 1)
Basic.publish(channel, "my_exchange", "my_queue", "4", persistent: true, priority: 0)
Basic.publish(channel, "my_exchange", "my_queue", "5", persistent: true, priority: 4)

Consumer:

def init(_) do
Connection.open(Application.get_env(:app, :rabbitmq))
{:ok, channel} = Channel.open(conn)
:ok = Basic.qos(channel, prefetch_count: 1, global: true)
{:ok, _consumer_tag} = Basic.consume(channel, "my_queue", nil, arguments: [{"x-priority", :signedint, 100}])
{:ok, %{channel: channel}}
end

def handle_info({:basic_deliver, payload, %{delivery_tag: tag, priority: priority}}, %{channel: channel} = state) do

Logger.info("Received message with payload: #{payload} and priority: #{priority}")

Basic.ack(channel, tag)

{:noreply, state}
end

After publishing, I run consumer.

Expected output:

Received message with payload: 5 and priority: 4
Received message with payload: 2 and priority: 3
Received message with payload: 1 and priority: 2
Received message with payload: 3 and priority: 1
Received message with payload: 4 and priority: 0

Actual output:

Received message with payload: 1 and priority: 2
Received message with payload: 2 and priority: 3 
Received message with payload: 3 and priority: 1
Received message with payload: 4 and priority: 0
Received message with payload: 5 and priority: 4

Anything I did wrong? Is message's priority not working?

Answer 1

As pointed out here, the correct queue parameter is x-max-priority. You are using x-priority, so your queue is not being declared as a priority queue.

Docs: https://www.rabbitmq.com/priority.html

Answer 2

Just to eliminate any doubt, it appears as though your consumer and producer are both connected and processing at the same time. When this happens, messages are routed directly to the consumer (essentially) without being queued.

So, RabbitMQ is working correctly, but your expectations need to be adjusted a bit. Message priority will only operate on messages in the queue, and even then, there are cases (e.g. parallel consumers) where priority may not be respected.

Bottom line - don't write a unit test that requires messages to be delivered in any particular order, as that is not a guarantee provided by message queuing systems.

Answer 3

One should not expect messages to be sorted inside a queue, due to what “queue” basically means (roughly any queue is either LIFO or FIFO, RabbitMQ’s one is the latter.)

Shuffling messages in the queue to sort them all by their priorities on every single insert would drastically decrease a performance. What is actually allowed by RabbitMQ (effective since 3.5.0,) is:

• prioritizing queues and
• prioritizing consumers.

NB Despite the above, starting with 3.5.0 RabbitMQ allows the sorting of messages that are stuck in the queue, assuming consumers get messages slower that they get to the queue. In such a case unconsumed messages will get sorted.

It still does not get to guaranteed sorting, though. Priorities only allow you to make some queues/consumers to “raise their hands” to receive messages first unless they are blocked. Otherwise, the next one in the priority chain would receive a message.

If you need to sort an income (it’s hard to imagine why would you ever pick up a message queue in such a case, but still,) you have to sort it either before sending to queue in publisher, or after collecting all the expected messages from the queue in consumer.