CODEWITHSUNDEEP
javarabbitmqspring-amqpspring-rabbit
ssnyder
ssnyder

Reputation: 192

Spring AMQP Rabbit - single consumer listening to multiple queues - what is the delivery order?

Let me first make sure I explain the problem:

I have to process multiple queues that are filled, in the order they were actually created. This means that if the queues contain:

q1: m1, m2, m5, m7

q2: m3, m6, m9

q3: m4, m8

I would like to process them so that m4 does not process before m1, m2, or m3. m3 can execute anytime (it does not have to wait for m1 and m2, but it is ok for it to wait since it will most likely be simpler/safer to implement). and m8 does not process until m7 is processed.

I know it would serialize the effort - but using multiple threads, and I already have locking on some other value in the payload helps ensure that they won't step on each other and gain some amount of parallel processing.

We just ran into issues that q3 processed before the records in q1 and q2, so it couldn't actually do what it was supposed to do. q1 and q2 do take longer to process, and we expect to have more records put into those queues as well.

I have requested that the sender change to a single queue, but I'm not sure they will be making this change (different team, different priorities), so I am trying to have a realistic backup plan.

Now here is my actual question: I've seen that you can have 1 listener for multiple queues - is there any documentation on the order in which I would receive the messages? Is it just a round robin, always taking the oldest record from each queue? Or is it always the oldest record from all queues it is listening to that is delivered to my listener?

Upvotes: 3

Views: 2450

Answers (1)

Gary Russell
Gary Russell

Reputation: 174554

It depends on the prefetch, by default, the prefetch is 1, which means the broker will deliver 1 message and wait for an ack. The prefetch applies to the channel (across all the queues).

If the container concurrentConsumers is 1 (default), they will be processed serially, but the order is indeterminate - it depends on how the broker delivers them. I don't know the internal algorithm used by rabbitmq when a single channel has consumers on multiple queues; it's best to assume it's indeterminate.

I have requested that the sender change to a single queue,

A producer publishes to an exchange with a routing key - he shouldn't care about the queue topology downstream. The consumer decides the queue topology by binding to that exchange - if you change the exchange to a fanout you can bind a single queue to it and you'll get messages in order, regardless of the routing key the producer uses.

If the producer "owns" the exchange and won't change it, you can bind a fanout exchange to his exchange and bind your single queue to that.

Of course, if he adds queues to his exchanges, messages will accumulate there.

But, as I said, producers need not be involved in the queue topology.

Upvotes: 3

Related Questions