CODEWITHSUNDEEP
rabbitmqhigh-availabilitypolicies
Pablo Pinargote
Pablo Pinargote

Reputation: 21

RabbitMQ queue-master-locator does not work when stoping one node

we are exploring the queue-master-locator 'min-masters' policy, and it looks like it works OK when we create new Queues, but when for some reason we need to stop one node of the cluster, every queues existing on that node get promoted (master) to the same node, for example:

Node A has 30 queues Node B has 0 queues Node C has 2 queues Node D has 3 queues

When we stop Node A, all 30 queues get promoted to Node B, it is that the expected result?. We was hoping that the 30 queues would be distribuited across Node B, C, and D...

Please we are starting to get crazy arround this. Has everyone experimented this scenario? and It is posible to achieve what we are expecting to happen when shutdown Node A in some automatic way?

Our policy is defined as follow:

Listing policies ... prod ha queues ^ {"ha-mode":"exactly","ha-params":3,"ha-sync-mode":"automatic","queue-master-locator":"min-masters"}

We have 4 nodes in the cluster, 2 RAM nodes and 2 Disk Nodes.

The policy works just fine when we create new queues but it does nothing when we stop one node.

Thanks

Upvotes: 2

Views: 1860

Answers (2)

Luke Bakken
Luke Bakken

Reputation: 9637

I just tried out your specific scenario using the latest pre-release code for 3.6.13 and I can reproduce it. The issue is that RabbitMQ does not move existing queue masters to rebalance queues - it will only move queues whose master existed on the node that went down. I have opened this issue to address what you report here.

You can use this unsupported script to rebalance queue masters in your cluster. If you have issues running the script, please post to rabbitmq-users and I will see it.

Upvotes: 1

user8808265
user8808265

Reputation: 2053

According to the official documentation:

If the master fails […] the longest running mirror is promoted to master

And the reason just follows:

the assumption being that it is most likely to be fully synchronised with the master.

https://www.rabbitmq.com/ha.html

Upvotes: 1

Related Questions