How to resolve Jboss EAP 7.1.4 java.util.concurrent.TimeoutException: No invocation response received in 30000 milliseconds

Question

I am experiencing concurrent.Timeout Exception from the JBoss EAP 7.1.4 GA server when running the application. Unfortunately, this is not a constant issue. happening time to time in different functional flows.

Exception is:

javax.ejb.EJBException: java.util.concurrent.TimeoutException: No invocation response received in 30000 milliseconds
Caused by: javax.ejb.EJBException: java.util.concurrent.TimeoutException: No invocation response received in 30000 milliseconds
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:207)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)

... 17 common frames omitted
Caused by: java.util.concurrent.TimeoutException: No invocation response received in 30000 milliseconds
at org.jboss.ejb.client.EJBClientInvocationContext.lambda$awaitResponse$3(EJBClientInvocationContext.java:876)
at org.jboss.ejb.client.EJBClientInvocationContext$ThrowableResult.getResult(EJBClientInvocationContext.java:1132)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:567)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:503)
at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocationResult(RemotingEJBClientInterceptor.java:56)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:569)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:503)
at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocationResult(TransactionPostDiscoveryInterceptor.java:133)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:569)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:503)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocationResult(DiscoveryEJBClientInterceptor.java:114)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:569)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:503)
at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocationResult(NamingEJBClientInterceptor.java:78)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:569)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:503)

in my JBoss configuration using below settings:

Invocation timeout 30 seconds

Transaction timeout 300 seconds ( which is the default value in JBoss)

findings so far:

https://access.redhat.com/solutions/1223983

This solution says to upgrade JBoss 6.4, but I am already using 7.1.4 version

JBoss Bugs:

https://issues.jboss.org/browse/JBEAP-8553

https://issues.jboss.org/browse/JBEAP-12075

All bugs fixed with the versions less than the JBoss that I used.

But furthermore, I have noticed that this log is there when timeout comes in:

WARN  [com.arjuna.ats.arjuna] (Periodic Recovery) [] Transaction 0:ffffac18bb91:-672b1e8b:5c25eea9:3be85 has 1 heuristic participant(s)!
WARN  [com.arjuna.ats.jta] (Periodic Recovery) [] ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac18bb91:-672b1e8b:5c25eea9:3be85, node_name=1, branch_uid=0:ffffac18bb91:-672b1e8b:5c25eea9:3be89, subordinatenodename=null, eis_name=java:jboss/datasources/ix3 >, heuristic: TwoPhaseOutcome.HEURISTIC_HAZARD, product: Microsoft SQL Server/14.00.2002, jndiName: java:jboss/datasources/ix3 com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@7f64f479 >
WARN  [com.arjuna.ats.jta] (Periodic Recovery) [] XAResourceRecord restored heuristic instance: XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac18bb91:-672b1e8b:5c25eea9:3be85, node_name=1, branch_uid=0:ffffac18bb91:-672b1e8b:5c25eea9:3be89, subordinatenodename=null, eis_name=java:jboss/datasources/ix3 >, heuristic: TwoPhaseOutcome.HEURISTIC_HAZARD, product: Microsoft SQL Server/14.00.2002, jndiName: java:jboss/datasources/ix3 com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@7f64f479 >
WARN  [com.arjuna.ats.arjuna] (Periodic Recovery) [] Transaction 0:ffffac18bb91:-672b1e8b:5c25eea9:3be85 restored heuristic participant XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac18bb91:-672b1e8b:5c25eea9:3be85, node_name=1, branch_uid=0:ffffac18bb91:-672b1e8b:5c25eea9:3be89, subordinatenodename=null, eis_name=java:jboss/datasources/ix3 >, heuristic: TwoPhaseOutcome.HEURISTIC_HAZARD, product: Microsoft SQL Server/14.00.2002, jndiName: java:jboss/datasources/ix3 com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@7f64f479 >

Tried to do some research and find the problem, but so far no luck.

Never ending periodic recovery of heuristic participants

JBoss spamming XA Recovery warnings

Can anyone having experience related to this can provide some suggestions, please?

Note: Some of the stack traces in exception cannot be published here because of company related codes are there. Apologies for that

Answer 1

This problem was solved. it was because of below reasons.

1. Few of our servers (VMs) are hanged because of Microsoft windows update. the recent update was 2GB and it was hanged when installing. CPU and memory usage 100% most of the time and not allocation for other processing.

2. Another reason was our clustering configuration was misbehaved as one of the cluster node's requests were blocking from the server firewall. it was a configuration mistake as forgot to add firewall rules.

Once resolved both of the issues, the issue was not happening again.

If anyone experiencing this kind of problem, recommend verifying all the infrastructure fundamentals before go to the deep level of investigation.