Infinispan: java.lang.OutOfMemoryError: Java heap space

Question

I'm running a JBoss 7 ha cluster with 4 nodes across 2 servers. Since a few weeks, the following exception is thrown sporadically, sometimes 2 or 3 times day, then one 1 oder 2 times a week. It looks like that the chance that the exceptions is thrown, is higher when there a more users active.

 21:38:24,337 ERROR [org.infinispan.remoting.ReplicationQueue] (Scheduled-replicationQueue-thread-0) [, , ] ISPN000072: Failed replicating 3 elements in replication queueorg.infinispan.CacheException: java.lang.OutOfMemoryError: Java heap space
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:166) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:181) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.ReplicationQueueImpl.flush(ReplicationQueueImpl.java:159) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.ReplicationQueueImpl$1.run(ReplicationQueueImpl.java:112) [infinispan-core-5.1.0.BETA5.jar:]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_29]
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [:1.6.0_29]
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [:1.6.0_29]
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_29]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_29]
      at java.lang.Thread.run(Thread.java:662) [:1.6.0_29] Caused by: java.lang.OutOfMemoryError: Java heap space
      at org.infinispan.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:104) [infinispan-api-5.1.0.BETA5.jar:]
      at java.io.FilterOutputStream.write(FilterOutputStream.java:60) [:1.6.0_29]
      at java.io.FilterOutputStream.write(FilterOutputStream.java:108) [:1.6.0_29]
      at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:83) [jboss-marshalling-1.3.4.GA.jar:]
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:113) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:66) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:389) [infinispan-core-5.1.0.BETA5.jar:]
      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
      at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) [jboss-marshalling-1.3.4.GA.jar:]
      at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) [jboss-marshalling-1.3.4.GA.jar:]
      at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:59) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:91) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:57) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:84) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.marshallCall(CommandAwareRpcDispatcher.java:211) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:232) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:110) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:411) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:153) [infinispan-core-5.1.0.BETA5.jar:]
      ... 12 more

I've already tried to increase the JVM heap and permgen size in host.xml up to 8/16 GB without success.

 <jvms>
      <jvm name="default">
           <heap size="16g" max-size="16g"/>
           <permgen size="8g" max-size="8g"/>
      </jvm>
 </jvms>

Any ideas? Or is it possible to increase the Infispan heap size?

Answer 1

This topic is already a few months old, but I was able to resolve this problem myself.

The issue was related to a specific application that stores data inside the user's session. Under some special and very rare circumstances, the amount of data was still too much and the time for serialization and de-serialization increased. For example, a normal user session had 10 custom attributes added, the sessions causing the problem had more than 10000 attributes added.

The solution (in my case!) was not to store this attributes inside the user's session anymore but to add them to a distributed ehcache.

This tool was a great help for discovering the problem: http://messadmin.sourceforge.net/

Answer 2

There is a memory leak (most likely) or a huge invocation happening.

Stacktrace in case of memory leak may be not related to the memory leak. It shows just what is a straw that broke the camel's back.

Adding JVM arg of "-XX:+HeapDumpOnOutOfMemoryError" to get a heap dump may help to find the real cause.

Also Perm Gen may be harmlessly decreased from 8gb to few hundreds of MBs, it is not related to out of memory error that the system experiences.