CODEWITHSUNDEEP
rabbitmq
Steve
Steve

Reputation: 1615

RabbitMQ keeps too many queue mirrors after rolling upgrade

We have upgraded our RabbitMQ cluster recently to the latest version with a rolling upgrade (remove one node after the other, upgrade it and then add it back to the cluster with the newer version).

Roughly speaking, we proceeded as follows:

# Put host into maintenance mode
rabbitmq-upgrade drain

# Install new rabbitmq version

# Re-join cluster
rabbitmqctl join_cluster rabbit@rabbit01

Regarding the rolling upgrade, we have deliberately not included all the details here because this process is not the problem. The problem is rather the following: We wanted to make sure that the queue leader replicas are evenly distributed again afterwards. So we did:

# Rebalance queues
rabbitmq-queues rebalance "all" --vhost-pattern ".*" --queue-pattern ".*"

But now we have a situation where some queues have too many mirrors! We are using classic queue mirroring with the following policy:

enter image description here

But the next image shows that some queues have +2 mirrors. Normally this should only state +1.

enter image description here

Of course you could say that this is nothing too bad. But since we use automated checks to make sure that for each queue the correct number of mirors exists, these tests are now failing. We are looking for a way to get RabbitMQ to clean up this situation again, so that only the configured number of mirrors is created and continued.

Upvotes: 0

Views: 621

Answers (1)

Lennart Lamerz
Lennart Lamerz

Reputation: 36

It is possible to apply a new policy to the queue, with the same values as the old, but with a higher priority and a different name. When the policy changes, RabbitMQ will apply the correct values and the mirrors that are to much will vanish. In your example the command with rabbitmqctl would look like this:

rabbitmqctl set_policy --vhost customers --priority 1 new-tmp-policy "^ha.sample_queue_" '{"ha-mode": "exactly", "ha-params": 2, "ha-sync-mode": "automatic", "queue-master-locator": "min-masters", "queue-mode": "lazy"}'

This will change the policy for the sample_queues to "new-tmp-policy" and the mirrors will vanish. Now just delete the "new-tmp-policy" and the old policy will be applied again for the sample_queues.

Upvotes: 2

Related Questions