Recommended Infinispan cache mode for Spring Session

Question

My app uses Infinispan's Spring Session implementation via the Infinispan Spring Boot Starter. I am using the embedded mode via InfinispanEmbeddedHttpSessionConfiguration. The Spring Session integration documentation is pretty sparse, and it doesn't mention anything about the recommended cache mode for the session cache.

I have been using the DIST_SYNC mode, but I notice occasional ConcurrentModificationException errors in the log. Is there a more appropriate cache mode, or could this be a bug?

I am using Spring Boot 2.2.10, Infinispan 9.4.20 and Spring Session 2.2.0. The app runs under JDK 11.

Edit

The exception I observe in the logs is below. I've put xxx. in place of my company's package names for our filters.

The exception occurs when the same session is reused in quick succession, in particular when our external security vendor runs weekly scans on our app. The exception does not occur every week. When it does occur, it leads to an eventual OutOfMemory error on the node that did not get the ConcurrentModificationException. The heap is full of TrianglePerCacheInboundInvocationHander$1 objects.

I have tried to reproduce the error in our development environment by running concurrent requests for 3 days, but to no avail.

I've looked at the Infinispan Spring Session code, and I see that each request gets its own instance of MapSession, materialised from the serialised data stored in Infinispan. I don't see how multiple concurrent requests for the same session could cause the ConcurrentModificationException, and I'm not aware of anywhere in my app that would allow concurrent access to a MapSession instance from multiple threads.

04:46:25.758 ERROR o.i.r.r.RpcManagerImpl          - ISPN000073: Unexpected error while replicating
java.util.ConcurrentModificationException: null
    at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1493)
    at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1526)
    at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1524)
    at org.infinispan.commons.marshall.MarshallUtil.marshallMap(MarshallUtil.java:52)
    at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:63)
    at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:31)
    at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
    at org.infinispan.marshall.core.GlobalMarshaller.writeInternalClean(GlobalMarshaller.java:643)
    at org.infinispan.marshall.core.GlobalMarshaller.lambda$findWriter$0(GlobalMarshaller.java:369)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:137)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1082)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1040)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
    at org.infinispan.marshall.core.ExternalJBossMarshaller.objectToObjectStream(ExternalJBossMarshaller.java:36)
    at org.infinispan.marshall.core.GlobalMarshaller.writeRawUnknown(GlobalMarshaller.java:617)
    at org.infinispan.marshall.core.GlobalMarshaller.writeUnknown(GlobalMarshaller.java:612)
    at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:412)
    at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
    at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
    at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:125)
    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:71)
    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:118)
    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:114)
    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:65)
    at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:638)
    at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:402)
    at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:355)
    at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:183)
    at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:176)
    at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:305)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1031)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1217)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:276)
    at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:390)
    at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490)
    at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:443)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:82)
    at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.localPrimaryOwnerWrite(TriangleDistributionInterceptor.java:422)
    at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.handleSingleKeyWriteCommand(TriangleDistributionInterceptor.java:392)
    at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.visitPutKeyValueCommand(TriangleDistributionInterceptor.java:113)
    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:55)
    at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:23)
    at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25)
    at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:70)
    at org.infinispan.interceptors.InvocationStage.thenApply(InvocationStage.java:45)
    at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:225)
    at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:734)
    at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:337)
    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:151)
    at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lambda$nonTxLockAndInvokeNext$1(AbstractLockingInterceptor.java:297)
    at org.infinispan.interceptors.SyncInvocationStage.addCallback(SyncInvocationStage.java:42)
    at org.infinispan.interceptors.InvocationStage.andHandle(InvocationStage.java:65)
    at org.infinispan.interceptors.locking.AbstractLockingInterceptor.nonTxLockAndInvokeNext(AbstractLockingInterceptor.java:292)
    at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:128)
    at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:40)
    at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:82)
    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:184)
    at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:309)
    at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252)
    at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:96)
    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:151)
    at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:220)
    at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:182)
    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:55)
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
    at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:60)
    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:124)
    at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
    at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:60)
    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
    at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
    at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
    at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1915)
    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1430)
    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:2040)
    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1415)
    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:235)
    at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:116)
    at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:195)
    at org.infinispan.spring.common.provider.SpringCache.put(SpringCache.java:159)
    at org.infinispan.spring.common.session.AbstractInfinispanSessionRepository.updateTTL(AbstractInfinispanSessionRepository.java:112)
    at org.infinispan.spring.common.session.AbstractInfinispanSessionRepository.lambda$getSession$1(AbstractInfinispanSessionRepository.java:105)
    at java.base/java.util.Optional.map(Optional.java:265)
    at org.infinispan.spring.common.session.AbstractInfinispanSessionRepository.getSession(AbstractInfinispanSessionRepository.java:105)
    at org.infinispan.spring.common.session.AbstractInfinispanSessionRepository.findById(AbstractInfinispanSessionRepository.java:79)
    at org.infinispan.spring.common.session.AbstractInfinispanSessionRepository.findById(AbstractInfinispanSessionRepository.java:32)
    at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getRequestedSession(SessionRepositoryFilter.java:351)
    at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getSession(SessionRepositoryFilter.java:289)
    at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getSession(SessionRepositoryFilter.java:192)
    at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:244)
    at xxx.MaxSimultaneousRequestsFilter.getKey(MaxSimultaneousRequestsFilter.java:266)
    at xxx.MaxSimultaneousRequestsFilter.doFilter(MaxSimultaneousRequestsFilter.java:109)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at xxx.LoginFilter.doFilter(LoginFilter.java:49)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at xxx.BigIPFilter.doFilter(BigIPFilter.java:41)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at xxx.EncodingFilter.doFilter(EncodingFilter.java:36)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141)
    at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at xxx.ExcludeSessionRepositoryFilter.doFilter(ExcludeSessionRepositoryFilter.java:78)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.jboss.marshalling.TraceInformation: null

Answer 1

Using the DIST_SYNC mode for sessions should be fine.

If a ConcurrentModificationException is thrown while serializing a session attribute, then it's a sign that the application is modifying an attribute value after inserting it in the session, and it should not do that.

If a ConcurrentModificationException happens while serializing the session itself, then it's likely a bug in the Infinispan Spring Session integration. Same if the ConcurrentModificationException is not related to serialization at all. I suggest posting the full stack trace in your question, or registering an account for the Infinispan Zulip Chat and posting the exception there.

As a final note, Infinispan 9.4.20 is pretty old, and an eventual bug fix is probably going to land only in Infinispan 12.x.