CODEWITHSUNDEEP
jbossdistributed-cachinginfinispan
Sohan Lal
Sohan Lal

Reputation: 143

Infinispan cluster fails to communicate in case of huge data

I am using Infinispan in Distributed Async mode with 4 nodes on 4 different systems. Each node runs with 3 GB of heap size.

Only one node plays the role of loader and tries to load 50 million records in chunks (in a loop where 5 million records go to cache 10 times). According to my calculation, 4 nodes can handle that much of data so space is not a problem.

When I start all 4 nodes, cluster forms successfully and data starts loading in the cache. But since the data is very huge, after sometime any one node is unable to get response from other one node and fails with below exception:

2013-11-01 05:35:14 ERROR org.infinispan.interceptors.InvocationContextInterceptor     - ISPN000136: Execution error
org.infinispan.util.concurrent.TimeoutException: Timed out after 15 seconds waiting for a response from INUMUU410-54463
 at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:459)
 at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:154)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534)

INUMUU410-54463 is the machine name.

Upvotes: 3

Views: 2562

Answers (1)

Radim Vansa
Radim Vansa

Reputation: 5888

(copied from Flavius comment above:)

What I'd do in your case is to split it to such that one putAll does not contain more than e.g. 1MB of data and then send these synchronously (using cache.getAdvancedCache().withFlags(FORCE_SYNCHRONOUS)). Or otherwise throttle the number of messages that are on air simlutaneously (see also the putAllAsync method on advanced cache).

Upvotes: 1

Related Questions