Infinispan async replication queue - not being used when configured

Question

Objective

I have a cluster using Infinispan in embedded + replication mode. Cluster size is just 2 systems. In order to examine the performance gains by using async-replication using replication-queue I experimented a bit whose details are as below

Following is the basic test setup which I am using

Cluster : simple 2 node cluster with following Infinispan configuration Code : I created one async cache by using config of "async_repl_cache" (refer infinispan.xml pasted below) as template-config and I DO NOT override any configuration as below

   templateCfg = ecManager.getCacheConfiguration("async_repl_cache");
   ecManager.defineConfiguration(cacheName, templateCfg);

Version : I use 5.3.0 of Infinispan in embedded mode

Infinispan Config XML

<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:6.0 http://www.infinispan.org/schemas/infinispan-config-5.3.xsd"
    xmlns="urn:infinispan:config:5.3">
  <global>
    <transport nodeName="${nodeName}">
      <properties>
        <property name="configurationFile" value="jgroups_tcp.xml"/>
      </properties>
    </transport>

  </global>

  <default>
    <clustering mode="replication">
        <sync/>
    </clustering>
  </default>

  <namedCache name="repl">
    <!-- Use the configuration of the default cache as it is -->
  </namedCache>

  <namedCache name="async_repl_cache">
    <jmxStatistics enabled="true"/>
    <clustering mode="replication">
        <async useReplQueue="true" replQueueInterval="2000" />
    </clustering>
  </namedCache>


</infinispan>

Observations

• I performed a small test with simple cache with 50K entries. One node just does puts and other node just receives the puts from remote system. This was done mainly to get a clear latency measurement differences in sync and async modes. What I noticed was that the duration for 50K puts remained the same

• Wrote a simple Btrace script to trace if add method of org.infinispan.remoting.ReplicationQueueImpl is ever being called so that I can be sure that queue is being used for replication remote calls. I noticed that this method is never invoked

• Used VisualVM's sampler to see if the call to put method could be traced to find its way to org.infinispan.remoting.ReplicationQueueImpl. I observed that entire call stack of RPC was being made in caller's thread instead of scheduled-replication thread

Question

I infer that replication is happening synchronously despite using async configuration. So, could someone please let me know if I am missing something in the configuration ?

PS : Not sure if this is something related - but ... I could see that the state-transfer happens properly on the node which joins a bit later after first node starts and receives all the data without any inconsistencies

Answer 1

(This would be more appropriate as a comment, but I don't have enough reputation.)
From your description I have a feeling that your test looks like this:

1. start 1st node
2. put the data into 1st node
3. start 2nd node

This will result into state transfer from 1st to 2nd node. Replication queue would only be used during step 2, if 2nd node was already running.
So my suggestion would be to first start both nodes, and then do the puts.