WildFly hung with many threads blocked in arjuna ThreadUtil.getThreadId

Question

we are facing an issue every 2-3 days on a WildFly server which uses a ScheduledThreadPoolExecutor to execute some tasks in a transactions.

When this happens, the CPU reaches 100% and the server is not responsive any more.

From the Thread dump, I have noticed that we have about 100 Threads blocked on the getThreadId of the arjuna Transaction Manager. We are using WildFly 16. Is it a known issue which is maybe solved in a newer release? Any other suggestion, besides increasing the arjuna log level ?

java.lang.Thread.State: BLOCKED (on object monitor)
at com.arjuna.ats.arjuna.utils.ThreadUtil.getThreadId(ThreadUtil.java:71)
- waiting to lock <0x0000000542a53950> (a java.lang.Class for com.arjuna.ats.arjuna.utils.ThreadUtil)
at com.arjuna.ats.arjuna.coordinator.BasicAction.addChildThread(BasicAction.java:610)
- locked <0x000000071bc00b68> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
at com.arjuna.ats.internal.arjuna.thread.ThreadActionData.pushAction(ThreadActionData.java:93)
at com.arjuna.ats.internal.arjuna.thread.ThreadActionData.pushAction(ThreadActionData.java:69)
at com.arjuna.ats.arjuna.AtomicAction.begin(AtomicAction.java:124)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.<init>(TransactionImple.java:104)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.begin(BaseTransaction.java:98)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.begin(BaseTransactionManagerDelegate.java:78)
at org.wildfly.transaction.client.provider.jboss.JBossLocalTransactionProvider.createNewTransaction(JBossLocalTransactionProvider.java:128)
at org.wildfly.transaction.client.LocalTransactionContext.beginTransaction(LocalTransactionContext.java:188)
at org.wildfly.transaction.client.ContextTransactionManager.begin(ContextTransactionManager.java:62)
at org.wildfly.transaction.client.ContextTransactionManager.begin(ContextTransactionManager.java:54)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:233)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:388)
at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.BasicComponentInstance.destroy(BasicComponentInstance.java:125)
at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.destroy(StatelessSessionComponent.java:69)
at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.destroy(StatelessSessionComponent.java:61)
at org.jboss.as.ejb3.pool.AbstractPool.doRemove(AbstractPool.java:80)
at org.jboss.as.ejb3.pool.AbstractPool.destroy(AbstractPool.java:69)
at org.jboss.as.ejb3.pool.strictmax.StrictMaxPool.stop(StrictMaxPool.java:171)
at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent.done(StatelessSessionComponent.java:120)
at org.jboss.as.ejb3.component.EJBComponent.stop(EJBComponent.java:605)
at org.jboss.as.ee.component.ComponentStartService$2.run(ComponentStartService.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:485)

Thanks!

Answer 1

The stack says that the thread is "... waiting to lock <0x0000000542a53950>". Can you find out which thread is holding that lock:

• find the process id using the jps command
• dump the stack for each thread using the jstack <process id> command

The thread dumps should indicate which thread is holding the lock and that information will hopefully give some indication of why that thread isn't making progress and not releasing the lock.