L2 Cache Deadlock Prevention

[kalinchan]

EclipseLink L2 Caching NullPointer Exception triggers a deadlock

The code that triggers the error in question is this line of code from the ObjectChangeSet class. This class seems to be responsible for synchronizing data that has been broadcasted to the current instance’s L2 cache:

protected Object getObjectForMerge(MergeManager mergeManager, AbstractSession session, Object primaryKey, ClassDescriptor descriptor) {
    ...
    CacheKey cacheKey = session.getIdentityMapAccessorInstance().getCacheKeyForObject(primaryKey, descriptor.getJavaClass(), descriptor, true);
    if (cacheKey != null) {
        if (cacheKey.acquireReadLockNoWait()) {
            domainObject = cacheKey.getObject();
            cacheKey.releaseReadLock();
        } else {
            if (!mergeManager.isTransitionedToDeferredLocks()) {
                session.getIdentityMapAccessorInstance().getWriteLockManager().transitionToDeferredLocks(mergeManager);
            }
            //1 - LOCK ACQUIRED
            cacheKey.acquireDeferredLock();
            domainObject = cacheKey.getObject();
            int tries = 0;
            while (domainObject == null) {
                ++tries;
                if (tries > MAX_TRIES){
                    //NPE triggered here - session.getParent is null
                    session.getParent().log(SessionLog.SEVERE, SessionLog.CACHE, "entity_not_available_during_merge", new Object[]{descriptor.getJavaClassName(), cacheKey.getKey(), Thread.currentThread().getName(), cacheKey.getActiveThread()});
                    break;
                }
                synchronized (cacheKey) {
                        if (cacheKey.isAcquired()) {
                            try {
                                cacheKey.wait(10);
                            } catch (InterruptedException e) {
                                //ignore and return
                            }
                        }
                        domainObject = cacheKey.getObject();
                    }
                }
                //2 - LOCK RELEASED
                cacheKey.releaseDeferredLock();
            }

It seems that session.getParent() can be null in some specific contexts, as EclipseLink’s AbstractSession class method does return null explicitly until overridden in a child class:

/**
  * INTERNAL:
  * Gets the parent session.
  */
public AbstractSession getParent() {
    return null;
}

EclipseLink L2 Caching Deadlock Detection triggers a NullPointerException, causing deadlocks

The issue has been reported by a customer who experienced multiple NPE events in the following stack trace:

java.lang.NullPointerException
at org.eclipse.persistence.internal.identitymaps.CacheKey.hashCode(CacheKey.java:406)
at java.util.HashMap.hash(HashMap.java:340)
at java.util.HashMap.containsKey(HashMap.java:597)
at org.eclipse.persistence.internal.helper.ConcurrencyUtil.get(ConcurrencyUtil.java:1572)
at org.eclipse.persistence.internal.helper.ConcurrencyUtil.enrichMapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKeyInfoAboutReadLocks(ConcurrencyUtil.java:1516)
at org.eclipse.persistence.internal.helper.ConcurrencyUtil.createConcurrencyManagerState(ConcurrencyUtil.java:722)
at org.eclipse.persistence.internal.helper.ConcurrencyUtil.dumpConcurrencyManagerInformationStep01(ConcurrencyUtil.java:543)
at org.eclipse.persistence.internal.helper.ConcurrencyUtil.dumpConcurrencyManagerInformationIfAppropriate(ConcurrencyUtil.java:477)
at org.eclipse.persistence.internal.helper.ConcurrencyUtil.determineIfReleaseDeferredLockAppearsToBeDeadLocked(ConcurrencyUtil.java:170)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:661)
at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:472)
...
The problem is that the error is caused by this piece of code that is tasked with releasing a deferred lock:

try {
    // Add debug metadata to concurrency manager state
    // The current thread will now be waiting for other threads to build the object(s) it could not acquire
    if(!currentThreadRegisteredAsWaitingForisBuildObjectOnThreadComplete) {
        currentThreadRegisteredAsWaitingForisBuildObjectOnThreadComplete = true;
        THREADS_WAITING_TO_RELEASE_DEFERRED_LOCKS.add(currentThread);
    }
    Thread.sleep(20);
    //Detection of deadlocks occur
    ConcurrencyUtil.SINGLETON.determineIfReleaseDeferredLockAppearsToBeDeadLocked(this, whileStartTimeMillis, lockManager, readLockManager, ConcurrencyUtil.SINGLETON.isAllowInterruptedExceptionFired());
} catch (InterruptedException interrupted) { 
    //NPE aren't caught here so the locks are never removed!!
    THREADS_WAITING_TO_RELEASE_DEFERRED_LOCKS.remove(currentThread);
    AbstractSessionLog.getLog().logThrowable(SessionLog.SEVERE, SessionLog.CACHE, interrupted);
    releaseAllLocksAcquiredByThread(lockManager);
    releaseAllLocksAquiredByThreadAlreadyPerformed = true;
    clearJustificationWhyMethodIsBuildingObjectCompleteReturnsFalse();
    throw ConcurrencyException.waitWasInterrupted(interrupted.getMessage());
}