ActiveMQ Artemis migration to the pluggable quorum configuration

Question

I have question about ActiveMQ Artemis cluster migration to the pluggable quorum configuration.

Currently we have a cluster in the test environment which has 6 servers (3 pairs of master and slave with classic replication), and I plan to migrate to the cluster with the pluggable quorum voting. The version of Artemis is 2.23.1.

I have configured another (pretest) cluster with 3 zookeeper nodes and 2 nodes of primary/backup Artemis. It seems to work well, but it is a pretest environment where we perform some experiments, and there are no clients and workload. So I have decided to reconfigure the test cluster to use pluggable quorum voting.

At first I thought that we can change role of each server from master to primary, and from slave to backup.

Previous configuration was - master:

<ha-policy>
  <replication>
    <master>
      <check-for-live-server>true</check-for-live-server>
      <vote-on-replication-failure>true</vote-on-replication-failure>
      <quorum-size>2</quorum-size>
      <group-name>group-for-each-pair</group-name>
    </master>
  </replication>
</ha-policy>

Slave:

<ha-policy>
  <replication>
    <slave>
      <allow-failback>true</allow-failback>
      <group-name>group-for-each-pair</group-name>
    </slave>
  </replication>
</ha-policy>

The group name is used for slave to determine to which master it has to connect to.

Unfortunately, this setting does not work in the primary and backup sections. I tried to configure it and get xsd validation error for broker.xml.

In the documentation there are some words about settings which are no longer needed in the pluggable quorum configuration:

There are some no longer needed classic replication configurations:

vote-on-replication-failure quorum-vote-wait vote-retries vote-retries-wait check-for-live-server

But there is nothing about <group-name>. Maybe it is a documentation issue.

New configuration is - primary:

<ha-policy>
  <replication>
    <primary>
        <manager>
           <class-name>org.apache.activemq.artemis.quorum.zookeeper.CuratorDistributedPrimitiveManager</class-name>
           <properties>
              <property key="connect-string" value="zookeeper-amq1:2181,zookeeper-amq2:2181,zookeeper-amq3:2181"/>
           </properties>
        </manager>
    </primary>
  </replication>
</ha-policy>

Backup:

<ha-policy>
  <replication>
    <backup>
        <manager>
           <class-name>org.apache.activemq.artemis.quorum.zookeeper.CuratorDistributedPrimitiveManager</class-name>
           <properties>
              <property key="connect-string" value="zookeeper-amq1:2181,zookeeper-amq2:2181,zookeeper-amq3:2181"/>
           </properties>
        </manager>
        <allow-failback>true</allow-failback>
     </backup>
  </replication>
</ha-policy>

When I tried start the cluster with these settings, I found that backup servers try to connect to any primary server, and some of them cannot start. And I have reverted back to the old configuration.

I read the documentation and found some settings which could help:

1. <coordination-id>. Used in multi-primary configuration and probably will not work in the section.
2. namespace in the Apache Curator settings. Maybe it can help to split servers to pairs where each backup will connect to it's primary in the same namespace. But it may be designed for another purpose (to have one zookeeper for the several separate clusters), and there could be some other problems.

Another option is to remove unnecessary 4 ActiveMQ Artemis servers and use only 1 pair of servers. It will require client reconfiguration, but clients will continue to work with only 2 remaining servers even if there are 6 servers remain in the connection string.

Is there a preferred way to migrate from classic replication to the pluggable quorum voting without changing cluster topology (6 servers)?

Any changes in this test environment (if succeeded) will be performed on the UAT and production clusters which have the same topology. So we would prefer a smooth migration if possible.

Answer 1

I recommend just using group-name as you were before. For example on the primary:

<ha-policy>
   <replication>
      <primary>
         <manager>
            <class-name>org.apache.activemq.artemis.quorum.zookeeper.CuratorDistributedPrimitiveManager</class-name>
            <properties>
               <property key="connect-string" value="zookeeper-amq1:2181,zookeeper-amq2:2181,zookeeper-amq3:2181"/>
            </properties>
         </manager>
         <group-name>group-for-each-pair</group-name>
      </primary>
   </replication>
</ha-policy>

And on the backup:

<ha-policy>
   <replication>
      <backup>
         <manager>
            <class-name>org.apache.activemq.artemis.quorum.zookeeper.CuratorDistributedPrimitiveManager</class-name>
            <properties>
               <property key="connect-string" value="zookeeper-amq1:2181,zookeeper-amq2:2181,zookeeper-amq3:2181"/>
            </properties>
         </manager>
         <group-name>group-for-each-pair</group-name>
         <allow-failback>true</allow-failback>
      </backup>
   </replication>
</ha-policy>

That said, I strongly encourage you to execute performance tests with a single HA pair of brokers. A single broker can potentially handle millions of messages per second so it's likely that you don't need a cluster of 3 primary brokers. Also, if your applications are connected to the cluster nodes such that messages are produced on one node and consumed from another then having a cluster may actually reduce overall message throughput due to the extra "hops" a message has to take. Obviously this wouldn't be an issue for a single HA pair.

Finally, dropping from 6 brokers down to 2 would significantly reduce configuration and operational complexity, and it's likely to reduce infrastructure costs substantially as well. This is one of the main reasons we implemented pluggable quorum voting in the first place.