ConcurrencyException in Eclipselink

Question

For a few months now, we have been experiencing a ConcurrencyException coming from Eclipselink (v2.7.10). The occurrence is very rare. It happens like a few times a week on a random instance out of hundred of instances. The exception looks as follows :

Caused by: org.eclipse.persistence.exceptions.ConcurrencyException: 
Exception Description: Wait was interrupted. 
Message: [Summary current cache key of thread Thread-2096 (ActiveMQ-client-global-threads) Stuck thread problem: unique tiny message number (98) 
 The Thread [Thread-2096 (ActiveMQ-client-global-threads)]  appears to be stuck (possible dead lock ongoing). 
 The thread is working in the context of (CacheKey) = (--- CacheKey  (ObjectNull. Most likely not yet in server session cache and in the process of being created.):  
 (primaryKey: 22,137,168,321) (object: null) (object hash code: 0) (cacheKeyClass: org.eclipse.persistence.internal.identitymaps.SoftCacheKey) (cacheKey hash code: 1194985224) 
 (current cache key owner/activeThread: Thread-1930 (ActiveMQ-client-global-threads)) (getNumberOfReaders: 0)  (concurrencyManagerId: 29,047,152) (concurrencyManagerCreationDate: 2024-12-03 09:00:08.616)  (totalNumberOfTimeCacheKeyAcquiredForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReadingBlewUpExceptionDueToCacheKeyHavingReachedCounterZero:  0)  (depth: 2) ---) . 
 The thread has been stuck for: (40,004 ms) 
  Bellow we will describe the ActiveLocks, DeferredLocks and ReadLocks for this thread. Summary of active locks owned by thread Thread-2096 (ActiveMQ-client-global-threads) Listing of all ACTIVE Locks.
Thread Name: Thread-2096 (ActiveMQ-client-global-threads) 
0 Active locks.
Summary of deferred locks (could not be acquired and cause thread to wait for object building to complete) of thread Thread-2096 (ActiveMQ-client-global-threads) Listing of all DEFERRED Locks.
Thread Name: Thread-2096 (ActiveMQ-client-global-threads) 
1 Deferred locks.
Deferred lock nr: 0 , Deferred cache key: --- CacheKey  (ObjectNull. Most likely not yet in server session cache and in the process of being created.):  (primaryKey: 22,137,168,321) (object: null) (object hash code: 0) (cacheKeyClass: com.company.technical.server.persistence.internal.cache.MonitoredSoftCacheKey) (cacheKey hash code: 1194985224) (current cache key owner/activeThread: Thread-1930 (ActiveMQ-client-global-threads)) (getNumberOfReaders: 0)  (concurrencyManagerId: 29,047,152) (concurrencyManagerCreationDate: 2024-12-03 09:00:08.616)  (totalNumberOfTimeCacheKeyAcquiredForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReadingBlewUpExceptionDueToCacheKeyHavingReachedCounterZero:  0)  (depth: 2) ---
Summary of read locks acquired by thread Thread-2096 (ActiveMQ-client-global-threads) Listing of all READ Locks. Step 001 - sparse summary loop over all read locks acquired:
Thread Name: Thread-2096 (ActiveMQ-client-global-threads) 
0 read locks. The lockManager for this thread is null. 
]
    at deployment.com.company.external//org.eclipse.persistence.exceptions.ConcurrencyException.waitWasInterrupted(ConcurrencyException.java:110)
    at deployment.com.company.external//org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:668)
    at deployment.com.company.external//org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:472)
    at deployment.com.company.external//org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5365)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1002)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInternal(ObjectBuilder.java:774)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:727)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:705)
    at deployment.com.company.external//org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:861)
    at deployment.com.company.external//org.eclipse.persistence.queries.Cursor.buildAndRegisterObject(Cursor.java:312)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.retrieveNextObject(ScrollableCursor.java:590)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.loadNext(ScrollableCursor.java:413)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.hasNext(ScrollableCursor.java:279)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.hasMoreElements(ScrollableCursor.java:268)

The exception shows that a CacheKey in shared cache has been locked by a thread that never released it. Any other thread, that tries to acquireDeferred/releaseDeferred on it, waits 40 seconds (timeout value) and blows up with ConcurrencyException. The workaround has been to either invalidate the shared cache or to restart the server instance. All attempts to reproduce in local environment have had no success.

There seemed to be a high correlation with the load on the instance and the activation of activation of isolated cache on one of the entities. We rolled back the activation of isolated cache, the issue didn't reappear for months and with re-activation, it was back. As isolated cache is something we would like to have in order to avoid contention issues, we have since come up with a workaround where at the end of the processing, a check is performed to see if there are any pending active locks held by the current thread and if so, release them explicitly :

Thread currentThread = Thread.currentThread();
DeferredLockManager deferredLockManager = ConcurrencyManager.getDeferredLockManager(currentThread);
if (deferredLockManager != null) {
    // Release active locks on the thread.
    if (!deferredLockManager.getActiveLocks().isEmpty()) {
        LOGGER.warn("DeferredLockManager has active locks : {}", getDetail(deferredLockManager));

        deferredLockManager.releaseActiveLocksOnThread();
    }
    // Remove the thread entry
    ConcurrencyManager.removeDeferredLockManager(currentThread);
}

The fix works well. We still see this happening a few times a week at most on a random instance.

Links from the web with similar issue:

https://www.eclipse.org/forums/index.php/t/1097290/

https://bugs.eclipse.org/bugs/show_bug.cgi?id=559307

Any pointers on the issue is much appreciated.