ActiveMQ Artemis connection-router config in a cluster

Question

I need to balance incoming connections on the live nodes in my ActiveMQ Artemis cluster using connection-router.

My cluster is made of 2 masters (masterA and masterB) and 2 slaves (slaveA and slaveB) split in 2 groups (group A and group B), that's a 2 live-backup pair. I'm using static cluster connections (tcp), the slave nodes are replicas of the master ones (ha-policy replication).

All works well before I try to add connection-router. HA cluster is okay with my base configuration (i.e. before trying to add connection-router).

Before I tried to add the connection-router config element, the slave nodes were presents in the static-connectors/connector-ref and there was no router parameter in the acceptor url.

My client app uses a ConnectionFactory URL containing both master nodes (slaves cannot accept connections until their master fails and failover happens so I won't put them here).

I have 3 instances of my client app, each capable of handling a single consumer, so what I expect is to have 1 or 2 consumer on each master, which I'm not able to achieve on every startup. Of course it happens sometimes, but that's just coincidence.

The client app uses the admin/admin username and password (for test purpose).

Here are my broker.xml of all my 4 instances (relevant sections only):

masterA

<connectors>
    <!-- Connector used to be announced through cluster connections and notifications -->
    <connector name="masterA">tcp://localhost:61616</connector>
    <connector name="slaveA">tcp://localhost:61617</connector>
    <connector name="masterB">tcp://localhost:61626</connector>
    <connector name="slaveB">tcp://localhost:61627</connector>
</connectors>

  <acceptors>

     <acceptor name="artemis">tcp://localhost:61616?router=router-cluster-connection;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor>

  </acceptors>

  <cluster-user>myclusteruser</cluster-user>

  <cluster-password>myclusterpsw</cluster-password>

  <cluster-connections>
    <cluster-connection name="my-cluster">
      <connector-ref>masterA</connector-ref>
      <check-period>25001</check-period>
      <connection-ttl>30001</connection-ttl>
      <initial-connect-attempts>-1</initial-connect-attempts>
      <reconnect-attempts>-1</reconnect-attempts>
      <use-duplicate-detection>true</use-duplicate-detection>
      <message-load-balancing>ON_DEMAND</message-load-balancing>
      <max-hops>2</max-hops>
      <notification-interval>500</notification-interval>
      <notification-attempts>100</notification-attempts>
      <static-connectors>
        <connector-ref>masterA</connector-ref>
        <connector-ref>masterB</connector-ref>
<!--            <connector-ref>slaveA</connector-ref>
        <connector-ref>slaveB</connector-ref>-->
      </static-connectors>
    </cluster-connection>
  </cluster-connections>

  <connection-routers>
    <connection-router name="router-cluster-connection">
      <key-type>USER_NAME</key-type>
      <key-filter>admin</key-filter>
      <local-target-filter>admin</local-target-filter>
      <policy name="LEAST_CONNECTIONS" />
      <pool>
        <username>admin</username>
        <password>admin</password>
        <quorum-size>2</quorum-size>
        <local-target-enabled>true</local-target-enabled>
        <cluster-connection>my-cluster</cluster-connection>
      </pool>
    </connection-router>
  </connection-routers>

  <ha-policy>
     <replication>
        <master>
          <group-name>grappe-a</group-name>
          <check-for-live-server>true</check-for-live-server>
          <initial-replication-sync-timeout>3600000</initial-replication-sync-timeout>
          <vote-on-replication-failure>true</vote-on-replication-failure>
        </master>
     </replication>
  </ha-policy>

slaveA

<connectors>
    <!-- Connector used to be announced through cluster connections and notifications -->
    <connector name="masterA">tcp://localhost:61616</connector>
    <connector name="slaveA">tcp://localhost:61617</connector>
    <connector name="masterB">tcp://localhost:61626</connector>
    <connector name="slaveB">tcp://localhost:61627</connector>
</connectors>


  <acceptors>

     <!-- Acceptor for every supported protocol -->
     <acceptor name="artemis">tcp://localhost:61617?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor>

  </acceptors>

  <cluster-user>myclusteruser</cluster-user>

  <cluster-password>myclusterpsw</cluster-password>

  <cluster-connections>
     <cluster-connection name="my-cluster">
      <connector-ref>slaveA</connector-ref>
      <check-period>25003</check-period>
      <connection-ttl>30003</connection-ttl>
      <initial-connect-attempts>-1</initial-connect-attempts>
      <reconnect-attempts>-1</reconnect-attempts>
      <use-duplicate-detection>true</use-duplicate-detection>
      <message-load-balancing>ON_DEMAND</message-load-balancing>
      <max-hops>2</max-hops>
      <notification-interval>500</notification-interval>
      <notification-attempts>100</notification-attempts>
      <static-connectors>
        <connector-ref>masterA</connector-ref>
        <connector-ref>masterB</connector-ref>
<!--            <connector-ref>slaveA</connector-ref>
        <connector-ref>slaveB</connector-ref>-->
      </static-connectors>
    </cluster-connection>
  </cluster-connections>

  <ha-policy>
     <replication>
        <slave>
          <group-name>grappe-a</group-name>
          <allow-failback>true</allow-failback>
          <initial-replication-sync-timeout>36000000</initial-replication-sync-timeout>
            </slave>
     </replication>
  </ha-policy>

masterB

<connectors>
    <!-- Connector used to be announced through cluster connections and notifications -->
      <connector name="masterA">tcp://localhost:61616</connector>
      <connector name="slaveA">tcp://localhost:61617</connector>
      <connector name="masterB">tcp://localhost:61626</connector>
      <connector name="slaveB">tcp://localhost:61627</connector>
</connectors>

  <acceptors>
     <!-- Acceptor for every supported protocol -->
     <acceptor name="artemis">tcp://localhost:61626?router=router-cluster-connection;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor>

  </acceptors>

  <cluster-user>myclusteruser</cluster-user>

  <cluster-password>myclusterpsw</cluster-password>

  <cluster-connections>
    <cluster-connection name="my-cluster">
      <connector-ref>masterB</connector-ref>
      <check-period>25002</check-period>
      <connection-ttl>30002</connection-ttl>
      <initial-connect-attempts>-1</initial-connect-attempts>
      <reconnect-attempts>-1</reconnect-attempts>
      <use-duplicate-detection>true</use-duplicate-detection>
      <message-load-balancing>ON_DEMAND</message-load-balancing>
      <max-hops>2</max-hops>
      <notification-interval>500</notification-interval>
      <notification-attempts>100</notification-attempts>
      <static-connectors>
        <connector-ref>masterA</connector-ref>
        <connector-ref>masterB</connector-ref>
<!--            <connector-ref>slaveA</connector-ref>
        <connector-ref>slaveB</connector-ref>-->
      </static-connectors>
    </cluster-connection>
  </cluster-connections>

  <connection-routers>
    <connection-router name="router-cluster-connection">
      <key-type>USER_NAME</key-type>
      <key-filter>admin|NULL</key-filter>
      <local-target-filter>admin|NULL</local-target-filter>
      <policy name="LEAST_CONNECTIONS" />
      <pool>
        <username>admin</username>
        <password>admin</password>
        <quorum-size>2</quorum-size>
        <local-target-enabled>true</local-target-enabled>
        <cluster-connection>my-cluster</cluster-connection>
      </pool>
    </connection-router>
  </connection-routers>

  <ha-policy>
     <replication>
        <master>
          <group-name>grappe-b</group-name>
          <check-for-live-server>true</check-for-live-server>
          <initial-replication-sync-timeout>3600000</initial-replication-sync-timeout>
          <vote-on-replication-failure>true</vote-on-replication-failure>
        </master>
     </replication>
  </ha-policy>

slaveB

<connectors>
    <!-- Connector used to be announced through cluster connections and notifications -->
    <connector name="masterA">tcp://localhost:61616</connector>
    <connector name="slaveA">tcp://localhost:61617</connector>
    <connector name="masterB">tcp://localhost:61626</connector>
    <connector name="slaveB">tcp://localhost:61627</connector>
</connectors>


  <acceptors>

     <!-- Acceptor for every supported protocol -->
     <acceptor name="artemis">tcp://localhost:61627?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor>

  </acceptors>

  <cluster-user>myclusteruser</cluster-user>

  <cluster-password>myclusterpsw</cluster-password>

  <cluster-connections>
     <cluster-connection name="my-cluster">
      <connector-ref>slaveB</connector-ref>
      <check-period>25004</check-period>
      <connection-ttl>30004</connection-ttl>
      <initial-connect-attempts>-1</initial-connect-attempts>
      <reconnect-attempts>-1</reconnect-attempts>
      <use-duplicate-detection>true</use-duplicate-detection>
      <message-load-balancing>ON_DEMAND</message-load-balancing>
      <max-hops>2</max-hops>
      <notification-interval>500</notification-interval>
      <notification-attempts>100</notification-attempts>
      <static-connectors>
        <connector-ref>masterA</connector-ref>
        <connector-ref>masterB</connector-ref>
<!--            <connector-ref>slaveA</connector-ref>
        <connector-ref>slaveB</connector-ref>-->
      </static-connectors>
    </cluster-connection>
  </cluster-connections>


  <ha-policy>
     <replication>
        <slave>
          <group-name>grappe-b</group-name>
          <allow-failback>true</allow-failback>
          <initial-replication-sync-timeout>36000000</initial-replication-sync-timeout>
        </slave>
     </replication>
  </ha-policy>

ActiveMQ Artemis version is 2.22.0, running on java 11.

How am I supposed to configure the connection-router in my brokers to evenly balance incoming connections?

Is it supported and a best practice to have HA on a broker doing connection-routing only?

If I add the connection-router element in the slave brokers, they fail to replicate. Is that expected? In case a master fails and failover happens, how can the connection-router feature still work if I can't add it on slaves if I want them to replicate?

I'll provide more details if needed.

Update The router relevant config elements

<connectors>
    <connector name="masterA">tcp://localhost:61616</connector>
    <connector name="slaveA">tcp://localhost:61617</connector>
    <connector name="masterB">tcp://localhost:61626</connector>
    <connector name="slaveB">tcp://localhost:61627</connector>
    <connector name="routerA">tcp://localhost:61620</connector>
</connectors>

<acceptors>
    <acceptor name="artemis">tcp://127.0.0.1:61620?router=router-cluster-connection;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor>
</acceptors>

<connection-routers>
    <connection-router name="router-cluster-connection">
        <key-type>USER_NAME</key-type>
        <key-filter>admin</key-filter>
        <policy name="ROUND_ROBIN" />
        <pool>
            <username>admin</username>
            <password>admin</password>
            <quorum-size>2</quorum-size>
            <local-target-enabled>false</local-target-enabled>
            <static-connectors>
                <connector-ref>masterA</connector-ref>
                <connector-ref>masterB</connector-ref>
            </static-connectors>
        </pool>
    </connection-router>
</connection-routers>

Answer 1

The LEAST_CONNECTIONS policy is good to evenly balance incoming connections but it has not been tested with HA (the replication issue due to backup brokers with connection routers should be fixed by ARTEMIS-4113).

Often, it is better to partition data across brokers than to balance incoming connections to improve performances, for further details see the symmetric-simple example.

To avoid a SPOF you could use a symmetric architecture, see the symmetric-redirect and the symmetric-simple examples