Artemis HA and cluster not working

Question

Below are setting of artemis cluster (3 servers) in broker.xml

     <!-- Clustering configuration -->

      <broadcast-groups>
         <broadcast-group name="my-broadcast-group">
            <broadcast-period>5000</broadcast-period>
            <jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
            <jgroups-channel>active_broadcast_channel</jgroups-channel>
            <connector-ref>netty-connector</connector-ref>
         </broadcast-group>
      </broadcast-groups>

      <discovery-groups>
         <discovery-group name="my-discovery-group">
            <jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
            <jgroups-channel>active_broadcast_channel</jgroups-channel>
            <refresh-timeout>10000</refresh-timeout>
         </discovery-group>
      </discovery-groups>

      <cluster-connections>
         <cluster-connection name="my-cluster">
            <connector-ref>netty-connector</connector-ref>
            <retry-interval>500</retry-interval>
            <use-duplicate-detection>true</use-duplicate-detection>
            <message-load-balancing>STRICT</message-load-balancing>
            <max-hops>1</max-hops>
            <discovery-group-ref discovery-group-name="my-discovery-group"/>
         </cluster-connection>
      </cluster-connections>

  <ha-policy>
         <shared-store>
            <colocated>
               <backup-port-offset>100</backup-port-offset>
               <backup-request-retries>-1</backup-request-retries>
               <backup-request-retry-interval>2000</backup-request-retry-interval>
               <max-backups>2</max-backups>
               <request-backup>true</request-backup>
               <master>
                  <failover-on-shutdown>true</failover-on-shutdown>
               </master>
               <slave>
                  <scale-down/>
               </slave>
            </colocated>
         </shared-store>
      </ha-policy>

Cluster and ha configuration are same in all servers. The failover scenario which i am trying to understand and execute is as below.

1. Start broker1,broker2,broker3 in sequence mentioned. Here I can see from admin UI that broker1 have backing up broker2 and broker3. broker2 have backing of broker1. broker3 does not have any backup.

2. I wrote below program to connect to server

   public static void main(final String[] args) throws Exception {
       Connection connection = null;
       InitialContext initialContext = null;
       try {
           Properties properties = new Properties();
           properties.put(Context.INITIAL_CONTEXT_FACTORY,
                   "org.apache.activemq.artemis.jndi.ActiveMQInitialContextFactory");
           properties.put("connectionFactory.ConnectionFactory",
                   "(tcp://localhost:61616,tcp://localhost:61617,tcp://localhost:61618)?ha=true&retryInterval=1000&retryIntervalMultiplier=1.0&reconnectAttempts=-1");
           properties.put("queue.queue/exampleQueue", "exampleQueue");
           // Step 1. Create an initial context to perform the JNDI lookup.
           initialContext = new InitialContext(properties);
   
           ConnectionFactory cf = (ConnectionFactory) initialContext.lookup("ConnectionFactory");
   
           // Step 2. Look-up the JMS Queue object from JNDI
           Queue queue = (Queue) initialContext.lookup("queue/exampleQueue");
           // Step 3. Create a JMS Connection
           connection = cf.createConnection("admin", "admin");
   
           // Step 4. Start the connection
           connection.start();
   
           // Step 5. Create a JMS session with AUTO_ACKNOWLEDGE mode
   
           Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
           // Step 8. Create a text message
           BytesMessage message = session.createBytesMessage();
           message.setStringProperty(InfoSearchEngine.QUERY_ID_HEADER_PARAM, "123");
           MessageConsumer consumer0 = session.createConsumer(queue);
           // Step 9. Send the text message to the queue
           while (true) {
               try {
                   Thread.sleep(500);
                   // Step 7. Create a JMS message producer
                   MessageProducer messageProducer = session.createProducer(queue);
                   messageProducer.send(message);
                   System.out.println("Sent message: " + message.getBodyLength());
               } catch (Exception e) {
                   System.out.println("Exception - " + e.getLocalizedMessage());
               }
           }
       } finally {
           if (connection != null) {
               // Step 20. Be sure to close our JMS resources!
               connection.close();
           }
           if (initialContext != null) {
               // Step 21. Also close the initialContext!
               initialContext.close();
           }
       }
   }
   

3. if I shutdown broker1, program diverts to broker2 and runs fine. If i shutdown broker2 then the program doest not connect to broker3.

I expected that broker3 should have started taking up the request since it was in cluster.

Answer 1

I can see from admin UI that broker1 have backing up broker2 and broker3. broker2 have backing of broker1. broker3 does not have any backup.

Failover in Artemis only works between a live and a backup. In your scenario broker1 is backing up broker2 so when you shutdown broker1 that means broker2 no longer has a backup so that when you shutdown broker2 no failover happens. You should specify <group-name> in your master and slave configurations so that your backups form in a more organized way so that this kind of situation doesn't happen.