CODEWITHSUNDEEP
replicationhornetq
Dag
Dag

Reputation: 10539

HornetQ replication fails, getting timeouts

I'm trying to get the new HornetQ 2.3.0.alpha replication running, on two different hosts. When I try it on my local machine it does work more or less (separated hornetq with different ports). I will explain it in more detail shortly.

I use the plain vanilla HornetQ 2.3.0.alpha on latest 64bit Xubuntu (local tests) and CentOs (distributed), Oracle JDK 6u33 (but also tried other versions).

Now when testing localy, I use a different port for the backup (5446). I start the live, then the backup server. The backup syncs with the live. I kill/stop the live. The backup will continue to work instead, and requests against the backup work (which now operates on 5445). Now I want to start the live again, which does not work, since the backup uses the 5445. Therefore my next test was to start the live/backup on separate machines.

On separated hosts, the live-server starts as expected. When I start the backup, it hangs when he copies some data directores. After that he gets a timeout. There is not firewall or anything else between them, I already checked that twice and with my local admin. Telnet'ing from the backup-host to the live-host works fine.

Here is the exception I get on the backup, I append my configuration below:

***********************************************************************************
java  -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -Xms512M -Xmx1024M -Dhornetq.config.dir=../config/stand-alone/non-clustered -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dlogging.configuration=../config/stand-alone/non-clustered/logging.properties -Djava.library.path=. -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jnp-client.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-twitter-integration.jar:../lib/hornetq-spring-integration.jar:../lib/hornetq-service-sar.jar:../lib/hornetq-rest.jar:../lib/hornetq-journal.jar:../lib/hornetq-jms.jar:../lib/hornetq-jms-client.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core.jar:../lib/hornetq-core-client.jar:../lib/hornetq-commons.jar:../lib/hornetq-bootstrap.jar:../config/stand-alone/non-clustered:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
***********************************************************************************
Unable to read the logging configuration from '../config/stand-alone/non-clustered/logging.properties' (java.net.MalformedURLException: no protocol: ../config/stand-alone/non-clustered/logging.properties)
11:50:37,611 INFO  [org.hornetq.integration.bootstrap] HQ101001: Starting HornetQ Server
11:50:38,932 INFO  [org.hornetq.core.server] HQ111001: backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=false,journalDirectory=../data/journal,bindingsDirectory=../data/bindings,largeMessagesDirectory=../data/large-messages,pagingDirectory=../data/paging)
11:50:38,950 WARN  [org.hornetq.core.server] HQ112216: Moving data directory ../data/bindings to ../data/bindings22
11:50:38,952 WARN  [org.hornetq.core.server] HQ112216: Moving data directory ../data/journal to ../data/journal22
11:50:38,953 WARN  [org.hornetq.core.server] HQ112216: Moving data directory ../data/paging to ../data/paging22
11:50:38,953 WARN  [org.hornetq.core.server] HQ112216: Moving data directory ../data/large-messages to ../data/large-messages22
11:51:39,106 ERROR [org.hornetq.core.server] HQ114002: Failure in initialisation: java.lang.RuntimeException: Could not estabilish the connection
    at org.hornetq.core.server.impl.HornetQServerImpl$SharedNothingBackupActivation.run(HornetQServerImpl.java:2152) [hornetq-core.jar:]
    at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]

java.lang.RuntimeException: Could not estabilish the connection
    at org.hornetq.core.server.impl.HornetQServerImpl$SharedNothingBackupActivation.run(HornetQServerImpl.java:2152)
    at java.lang.Thread.run(Thread.java:662)

Configuration files

I used the default configuration files as base (config/stand-alone/non-clustered)

Live Configuration files

hornetq-configuration.xml

<configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

   <paging-directory>${data.dir:../data}/paging</paging-directory>
   <bindings-directory>${data.dir:../data}/bindings</bindings-directory>
   <journal-directory>${data.dir:../data}/journal</journal-directory>
   <journal-min-files>10</journal-min-files>
   <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>

   <connectors>
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
      <connector name="remote-connector">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="192.168.40.155"/>
         <param key="port" value="5445"/>
      </connector>
   </connectors>

   <acceptors>
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="0.0.0.0"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
   </acceptors>

   <security-settings>
      <security-setting match="#">
         <permission type="createNonDurableQueue" roles="guest"/>
         <permission type="deleteNonDurableQueue" roles="guest"/>
         <permission type="consume" roles="guest"/>
         <permission type="send" roles="guest"/>
      </security-setting>
   </security-settings>

   <address-settings>
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>10485760</max-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
         <address-full-policy>BLOCK</address-full-policy>
      </address-setting>
   </address-settings>

   <shared-store>false</shared-store>
   <clustered>true</clustered>
   <failover-on-shutdown>false</failover-on-shutdown>
   <cluster-user>cluster-user</cluster-user>
   <cluster-password>cluster123</cluster-password>

   <cluster-connections>
      <cluster-connection name="test-cluster">
         <address>jms</address>
         <connector-ref>netty</connector-ref>
         <retry-interval>2000</retry-interval>
         <use-duplicate-detection>true</use-duplicate-detection>
         <forward-when-no-consumers>true</forward-when-no-consumers>
         <max-hops>1</max-hops>
         <static-connectors allow-direct-connections-only="true">
            <connector-ref>remote-connector</connector-ref>
         </static-connectors>
      </cluster-connection>
   </cluster-connections>

   <backup>false</backup>

</configuration>

hornetq-jms.xml

<configuration xmlns="urn:hornetq"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">

   <connection-factory name="NettyConnectionFactory">
      <xa>false</xa>
      <ha>true</ha>
      <reconnect-attempts>-1</reconnect-attempts>
      <retry-interval>1000</retry-interval>

      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="/ConnectionFactory"/>
      </entries>
   </connection-factory>

   <queue name="DLQ">
      <entry name="/queue/DLQ"/>
   </queue>

   <queue name="ExpiryQueue">
      <entry name="/queue/ExpiryQueue"/>
   </queue>

   <queue name="testing">
      <entry name="/queue/testing" />
      <durable>true</durable>
   </queue>

</configuration>

Backup Configuration files

hornetq-configuration.xml

<configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

   <paging-directory>${data.dir:../data}/paging</paging-directory>
   <bindings-directory>${data.dir:../data}/bindings</bindings-directory>
   <journal-directory>${data.dir:../data}/journal</journal-directory>
   <journal-min-files>10</journal-min-files>
   <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>

   <connectors>
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
      <connector name="remote-connector">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="192.168.40.180"/>
         <param key="port" value="5445"/>
      </connector>
   </connectors>

   <acceptors>
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="0.0.0.0"/>
         <param key="port"  value="5445"/>
      </acceptor>
   </acceptors>

   <security-settings>
      <security-setting match="#">
         <permission type="createNonDurableQueue" roles="guest"/>
         <permission type="deleteNonDurableQueue" roles="guest"/>
         <permission type="consume" roles="guest"/>
         <permission type="send" roles="guest"/>
      </security-setting>
   </security-settings>

   <address-settings>
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>10485760</max-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
         <address-full-policy>BLOCK</address-full-policy>
      </address-setting>
   </address-settings>

   <shared-store>false</shared-store>
   <clustered>true</clustered>
   <failover-on-shutdown>false</failover-on-shutdown>
   <cluster-user>cluster-user</cluster-user>
   <cluster-password>cluster123</cluster-password>
   <live-connector-ref connector-name="remote-connector"/>


   <cluster-connections>
      <cluster-connection name="test-cluster">
         <address>jms</address>
         <connector-ref>netty</connector-ref>
         <retry-interval>2000</retry-interval>
         <use-duplicate-detection>true</use-duplicate-detection>
         <forward-when-no-consumers>true</forward-when-no-consumers>
         <max-hops>1</max-hops>
         <static-connectors allow-direct-connections-only="true">
            <connector-ref>remote-connector</connector-ref>
         </static-connectors>
      </cluster-connection>
   </cluster-connections>

   <backup>true</backup>

</configuration>

hornetq-jms.xml

<configuration xmlns="urn:hornetq"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">

   <connection-factory name="NettyConnectionFactory">
      <xa>false</xa>
      <ha>true</ha>
      <reconnect-attempts>-1</reconnect-attempts>
      <retry-interval>1000</retry-interval>

      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="/ConnectionFactory"/>
      </entries>
   </connection-factory>

   <queue name="DLQ">
      <entry name="/queue/DLQ"/>
   </queue>

   <queue name="ExpiryQueue">
      <entry name="/queue/ExpiryQueue"/>
   </queue>

   <queue name="testing">
      <entry name="/queue/testing" />
      <durable>true</durable>
   </queue>

</configuration>

When I created the sandbox locally, it uses the same configuration, except that for backup the hornetq-beans.xml is changed to move the ports from 1099/1098 to 1199/1198.

I'm working on this now for days, and tried already various combinations, so please excuse me If I missed something. Thanks a lot!

Upvotes: 0

Views: 1272

Answers (1)

Clebert Suconic
Clebert Suconic

Reputation: 5383

HornetQ 2.3 is under active development. I would recommend you to ask this on the User's forum instead of SOF where all the developers work.

Anyways we are about to release a Beta where we changed how the connection happens between the nodes.. we even removed a paramter and changed a few things. So I would definitely recommend you to talk to us on the HOrnetQ user's forum.. and maybe give it a try on hornetq/master on github.

Upvotes: 1

Related Questions