JPA/Wildfly/MsSql: GenericJDBCException: Unable to acquire JDBC Connection

Question

currently my team and I are facing a strange problem. By now we have been spending 4 days searching for the solution.

We are developing a Java EE Web Application. We have been using Wildfly 14. Using Wildfly 14, there was no problem. Now we have upgraded Wildfly to Version 18 and afterwars to 19 and are now facing the following problem. As database we have an MS SQL.

The datasource looks like this:

<datasource jndi-name="java:/label" pool-name="Label">
                <connection-url>jdbc:sqlserver://localhost:1433;DatabaseName=label</connection-url>
                <driver>mssql-jdbc-8.2.2.jre8.jar</driver>
                <security>
                    <user-name>label</user-name>
                    <password>label</password>
                </security>
</datasource>

And the persistence.xml is the following:

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0"
xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">


<persistence-unit name="label-pu"
    transaction-type="JTA">
    <jta-data-source>java:/label</jta-data-source>
    <properties>
        <property name="hibernate.dialect"
            value="org.hibernate.dialect.SQLServerDialect" />
    </properties>
</persistence-unit>

We have a Bean in which we inject an EntityManager using the persistence-unit. When calling the bean's method we are getting an exception. The datasource is working. We checked that in the wildfly ui.

@Dependent
public class ConstructionService {

@PersistenceContext(unitName = "label-pu")
private EntityManager em;

public List<ConstructionRecordEntity> retrieveRecordForConstruction(int contructionId) {

    String sqlSelectEmployeeId = "SELECT lfdnr, " + //
            "                            erwkz1, " + //
            "                            erwbe, " + //
            "                            erwdat " + //
            "                     FROM [label].[dbo].[aderw] where adnr = " + contructionId;//

    return em//
            .createNativeQuery(sqlSelectEmployeeId, ConstructionRecordEntity.class)//
            .getResultList();
}

In the following you can see the stacktrace we are getting.

15:11:24,506 WARN  [org.camunda.bpm.engine.rest.exception] (default task-1) ENGINE-REST-HTTP500 javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
    at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
    at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1515)
    at org.hibernate.query.Query.getResultList(Query.java:132)
    at de._._.ConstructionService.retrieveRecordForConstruction(ConstructionService.java:26)

...

Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:109)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:136)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:50)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)
    at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2082)
    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2012)
    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1990)
    at org.hibernate.loader.Loader.doQuery(Loader.java:949)
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:351)
    at org.hibernate.loader.Loader.doList(Loader.java:2787)
    at org.hibernate.loader.Loader.doList(Loader.java:2770)
    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2604)
    at org.hibernate.loader.Loader.list(Loader.java:2599)
    at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:338)
    at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2243)
    at org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:1069)
    at org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:170)
    at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1506)
    ... 306 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@c18c68f[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@51667cf2 connection handles=0 lastReturned=1594905084472 lastValidated=1594904382991 lastCheckedOut=1594905084468 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@66e3a014 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@685b64a3[pool=Label] xaResource=LocalXAResourceImpl@671cf884[connectionListener=c18c68f connectionManager=bec01f1 warned=false currentXid=null productName=Microsoft SQL Server productVersion=15.00.2000 jndiName=java:/label] txSync=null]
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:159)
    at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
    at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
    at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:106)
    ... 325 more
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@c18c68f[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@51667cf2 connection handles=0 lastReturned=1594905084472 lastValidated=1594904382991 lastCheckedOut=1594905084468 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@66e3a014 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@685b64a3[pool=Label] xaResource=LocalXAResourceImpl@671cf884[connectionListener=c18c68f connectionManager=bec01f1 warned=false currentXid=null productName=Microsoft SQL Server productVersion=15.00.2000 jndiName=java:/label] txSync=null]
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:1055)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:792)
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
    ... 329 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:571)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:977)
    ... 331 more
Caused by: javax.transaction.SystemException: Error enlisting resource in transaction=Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a83801:34f7831:5f103dd7:8e1 status: ActionStatus.ABORT_ONLY >, owner=Local transaction context for provider JBoss JTA transaction provider)
    at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:957)
    at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:394)
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:564)
    ... 332 more
Caused by: java.lang.Throwable: Failed to enlist
    at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.enlist(TxConnectionListener.java:1001)
    at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:379)
    ... 333 more

I apreciate any hint.

Answer 1

We found out, that this is related to Camunda, when we first thought is was a pure Java EE Problem.

Using the mentioned Bean in a Rest Service works perfectly. Only when we inject the Bean into a Camunda JavaDelegate there is a problem.

We could solve this by adding

<system-properties>
   <property name="com.arjuna.ats.arjuna.allowMultipleLastResources" value="true"/>
</system-properties>

to Wildfly Standalone.

It is working but we are getting a warning. Actually, this is necessary if you are using xa-datasource to have a transaction that surrounds multiple datasources. However, we do not have xa-datasource and we do not need a transaction that surrounds everything.

As far as we understand Camunda uses JTA to make the process jobs transaction safe. So when we are making a db query inside the JavaDelegate, there is a transaction inside a transaction. The first transaction is instantiated by Camunda the second one is instantiated by us by using JPA to query the database. Nevertheless, I do not understand. This scenario should be very common. So other people should have experienced this behavior, too? Or maybe it is related to the MS SQL Database?

The other thing that I do not understand is, that using JPA inside JavaDelegate worked with camunda 7.10 and in the latest version 7.13 it stopped working.

So, we found a workaround and we are not stuck anymore. Nonetheless, this workaround is not satisfying as we do not understand entirely what is going on underneath and we keep looking for an appropriate solution.

Answer 2

I was working with WildFly server long time ago so there is just my amateur thought:

There is something bad happened with transaction. The issue point is connected with resource enlistement: http://javadox.com/org.jboss.ironjacamar/ironjacamar-core-impl/1.2.7.Final/org/jboss/jca/core/connectionmanager/listener/TxConnectionListener.TransactionSynchronization.html#enlist()

According to the documentation, the database connection is also an "Transactional Resource". So, yes, we do not have an database connection for some reason.

1. May be someone close transaction for some reason? Try to check transaction status with injection of UserTransaction or with debugging of TxConnectionListener class (see the log)
2. If transaction is ok, then just debug from the point of error. Try to find the check that leads to an error. You can find the proper library to debug inside WildFly directory (with Far Manager, Double Commander and so on) and connect it to the IDE to be able to make a break point in the proper place.