CODEWITHSUNDEEP
spring-bootcachingcluster-computinginfinispan
ecostanzi
ecostanzi

Reputation: 342

Infinispan clustered REPL_ASYNC cache: command indefinitely bounced between two nodes

Im running a spring boot application using infinispan 10.1.8 in a 2 node cluster. The 2 nodes are communicating via jgroups TCP. I configured several REPL_ASYNC.

The problem: One of these caches, at some point is causing the two nodes to exchange the same message over and over, causing high CPU and memory usage. The only way to stop this is to stop one of the two nodes.

More details, here is the configuration.

 org.infinispan.configuration.cache.Configuration replAsyncNoExpirationConfiguration = new ConfigurationBuilder()
                .clustering()
                    .cacheMode(CacheMode.REPL_ASYNC)
                .transaction()
                    .lockingMode(LockingMode.OPTIMISTIC)
                .transactionMode(TransactionMode.NON_TRANSACTIONAL)
                .statistics().enabled(cacheInfo.isStatsEnabled())
                .locking()
                    .concurrencyLevel(32)
                    .lockAcquisitionTimeout(15, TimeUnit.SECONDS)
                    .isolationLevel(IsolationLevel.READ_COMMITTED)
                .expiration()
                    .lifespan(-1) //entries do not expire
                    .maxIdle(-1) // even when they are idle for some time
                    .wakeUpInterval(-1) // disable the periodic eviction process
                .build();

One of these caches (named formConfig) is causing me abnormal communication between the two nodes, this is what happens:

  1. with jmeter I generate traffic load targeting only node 1
  2. for some time node 2 receives cache entries from node 1 via SingleRpcCommand, no anomalies, even formConfig cache behaves properly
  3. after some time a new cache entry is sent to the formConfig cache

At this point the same message seems to keep bouncing between the two nodes:

Some other things:

Here's a more complete log file including logs from both nodes.

Other infos:

I'm suspecting

Upvotes: 0

Views: 234

Answers (1)

pruivo
pruivo

Reputation: 1344

The only scenario where this can happen is when the SimpleKey has different hashCode().

Are there any exceptions in the log? Are you able to check if the hashCode() is the same after serialization & deserialization of the key?

Upvotes: 1

Related Questions