Reputation: 26
Mode Clustering ,Wildfly 21 , One node out of 2 nodes suddenly gets down
I am using Spring MVC 5, java 11, mod_cluster/1.3.16. Final. Suddenly 1 node fails out of 2 and the below exception prints on the server console. I am out of solution and need help.
2024-06-19 02:22:08,245 INFO [org.infinispan.CLUSTER] (thread-57,ejb,jvm1) ISPN100001: Node jvm3 left the cluster 2024-06-19 02:22:08,245 INFO [org.wildfly.clustering.ee.infinispan] (default task-21) WFLYCLEEINF0001: Failed to cancel WRQ0WOd-ioxzBQV6kiyhumj2RQBcyM73Wxyr05Wf on primary owner.: java.util.concurrent.CancellationException at [email protected]//org.wildfly.clustering.server.dispatcher.ServiceRequest.completeExceptionally(ServiceRequest.java:86) at [email protected]//org.jgroups.blocks.UnicastRequest.viewChange(UnicastRequest.java:78) at [email protected]//org.jgroups.blocks.RequestCorrelator.lambda$receiveView$1(RequestCorrelator.java:281) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3605) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at [email protected]//org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:281) at [email protected]//org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:228) at [email protected]//org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:569) at [email protected]//org.jgroups.JChannel.up(JChannel.java:765) at [email protected]//org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:911) at [email protected]//org.jgroups.protocols.FORK.up(FORK.java:132) at [email protected]//org.jgroups.protocols.FRAG3.up(FRAG3.java:151) at [email protected]//org.jgroups.protocols.FlowControl.up(FlowControl.java:338) at [email protected]//org.jgroups.protocols.pbcast.GMS.installView(GMS.java:707) at [email protected]//org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:65) at [email protected]//org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:113) at [email protected]//org.jgroups.protocols.pbcast.GMS.up(GMS.java:922) at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:330) at [email protected]//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297) at [email protected]//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:450) at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:952) at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.removeAndDeliver(NAKACK2.java:886) at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:860) at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:682) at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338) at [email protected]//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197) at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338) at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338) at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338) at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338) at [email protected]//org.jgroups.protocols.TP.passBatchUp(TP.java:1411) at [email protected]//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284) at [email protected]//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136) at [email protected]//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at [email protected]//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) at [email protected]//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70) at java. base/java.lang.Thread.run(Thread.java:829)
2 - web.xml contains <distributable/> tag
3- The Widlfy 21 standalone configuration is below.
<subsystem xmlns="urn:jboss:domain:infinispan:11.0">
<cache-container name="web" default-cache="routing" module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="sso">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
</replicated-cache>
<replicated-cache name="routing"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb" default-cache="routing" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="routing"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="hibernate" module="org.infinispan.hibernate-cache">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<heap-memory size="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="pending-puts">
<expiration max-idle="60000"/>
</local-cache>
<invalidation-cache name="entity">
<heap-memory size="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps"/>
</cache-container>
</subsystem>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<socket-protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="org.jgroups.protocols.TCPPING">
<property name="initial_hosts">10.20.30.101[7600],10.20.30.107[7600]</property>
<property name="port_range">0</property>
</protocol>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<subsystem xmlns="urn:jboss:domain:modcluster:5.0">
<proxy name="default" advertise="true" advertise-security-key="######" advertise-socket="modcluster" balancer="cslt1" listener="ajp" node-timeout="120" sticky-session="true">
<dynamic-load-provider>
<load-metric type="busyness"/>
</dynamic-load-provider>
</proxy>
</subsystem>
4 - Apache configuration is below.
<VirtualHost 10.20.30.80:9080>
<Directory />
Require ip 10.20.30.101
Require ip 10.20.30.107
</Directory>
ServerAdvertise On
AdvertiseGroup 224.0.1.105:23367
EnableMCPMReceive on
ManagerBalancerName cslt1
AdvertiseSecurityKey ######
<Location /mod_cluster-manager>
SetHandler mod_cluster-manager
#Require ip 10.20.30.80
</Location>
</VirtualHost>
Upvotes: 0
Views: 30