Titan Cassandra Multithreaded Transactions Locking

Question

I have created a class which wraps within it a Graph. For example:

public class GraphManager(){
    Graph graph;
    public GraphManager(Graph graph){
        this.graph = graph;
    }
    public void commitGraph(){
        graph.commit();
    }
}

This GraphManager allows me to interact with the graph in predefined ways. I construct this GraphManager using a factory:

public class GraphManagerFactory(){
    public static GraphManager getGraphManager(){
        return new GraphManager(TitanFactory.open("conf/titan-cassandra.properties"));
    }
}

That is the base framework. Now onto the problem, using a rest controller I receive a JSON file. This results in instantiating a GraphManager which translates the file into a graph and then commits it. The basic paradigm is as follows:

public class Controller(){
    public List<String> handleRequest(){
        GraphManager manager = GraphManagerFactory.getGraphManager();
        //Do some work with graph manager
        synchronised(Controller.class){
            manager.commitGraph();
        }
    }
}

With the code above I assure that only one thread can commit to the graph at any time. However despite that I still get a PermanentLockingException:

com.thinkaurelius.titan.diskstorage.locking.PermanentLockingException: Local lock contention
at com.thinkaurelius.titan.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:313) ~[titan-core-1.0.0.jar:na]
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:89) ~[titan-core-1.0.0.jar:na]
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:40) ~[titan-core-1.0.0.jar:na]
at com.thinkaurelius.titan.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:240) ~[titan-core-1.0.0.jar:na]
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:554) ~[titan-core-1.0.0.jar:na]
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:683) ~[titan-core-1.0.0.jar:na]
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:na]
at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263) [titan-core-1.0.0.jar:na]
at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94) [gremlin-core-3.0.2-incubating.jar:3.0.2-incubating]
at io.mindmaps.core.accessmanager.GraphAccessManagerImpl.commit(GraphAccessManagerImpl.java:811) [mindmaps-core-0.0.5-SNAPSHOT.jar:na]
at io.mindmaps.graphmanager.listener.TransactionController.commitGraph(TransactionController.java:98) [classes/:na]
at io.mindmaps.graphmanager.listener.TransactionController.validateAndCommit(TransactionController.java:84) [classes/:na]
at io.mindmaps.graphmanager.listener.TransactionController.loadData(TransactionController.java:66) [classes/:na]
at io.mindmaps.graphmanager.listener.TransactionController.lambda$postTransaction$0(TransactionController.java:43) [classes/:na]
at io.mindmaps.graphmanager.loader.QueueManager.handleJob(QueueManager.java:76) ~[classes/:na]
at io.mindmaps.graphmanager.loader.QueueManager.lambda$addJob$3(QueueManager.java:24) ~[classes/:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]

How can this occur when only one commit is allowed at a time ?

Answer 1

While the answer I accepted is 100% correct. I want to highlight more clearly what I did to escape from lock contentions (Much of this is based/thanks to the accepted answer):

Step 1: As recommend rather than wrap an instance of the graph I wrapped a new transaction in each GraphManager. i.e. I made the factory as follows:

public class GraphManagerFactory(){
    TitanGraph instance;
    public static GraphManager getGraphManager(){
        if(instance = null){
            instance = TitanFactory.open("conf/titan-cassandra.properties");
        }
        return new GraphManager(instance.newTransaction());
    }
}

This step lead to a lot of improvement. I still had lock contentions but they were resolved more quickly.

Step 2: When building the graph for the first time I also provided the schema in advance. Specifically rather than let titan build vertex properties and edges implicitly I built them explicitly before even adding the first vertex. This is a simple matter of using management.makeEdgeLabel(label).make(); for edge labels and management.makePropertyKey(label).dataType(String.class).make(); for vertex properties. An added benefit of this is that I can perform batch loading more easily. This means expanding the Factory again to be:

public class GraphManagerFactory(){
    TitanGraph instance;
    public static GraphManager getGraphManager(){
        if(instance = null){
            instance = TitanFactory.open("conf/titan-cassandra.properties");
            TitanManagement management = instance.openManagement();
            //Check if the labels exist before creating explicitly. 
            //If they don't exist do the following:
            management.makeEdgeLabel("EdgeLabel").make();
            management.makePropertyKey("property").dataType(String.class).make();
            management.commit();
        }
        return new GraphManager(instance.newTransaction());
    }
}

Step 3: The final step which removed contentions almost entirely was to increase the id block size to graph.configuration().setProperty("ids.block-size", 100000); . This final step may only be applicable to me though as I am performing large loading operations simultaneously.

Answer 2

First, I would recommend that you not create a TitanGraph instance per request (expensive). Create one TitanGraph and share it across requests. Next, you need to take great care with web applications that transaction do not leak between requests (a transaction is bound to the current thread). You can assure this by being sure that a request cleans up after itself by always issuing a rollback() or commit() when it completes (in error or success as necessary). You can doubly assure this by issuing a rollback() at the start of a new request.

With all that in mind, let's answer your question. Just because you have restricted the commit() action to a single thread doesn't prevent other threads from opening transactions. Another request handled by a different thread could quite easily try to grab a lock for the same key and be blocked on commit ending in the locking exception you're seeing.

"Transaction will eventually fail in sufficiently large systems." The PermanentLockingException must be treated as an expected side-effect of using locks and the typical approach to dealing with them is through retrying the entire transaction on encountering one. You should design your base architecture on that premise.

Some other tips in this area:

1. You should look to limit use of locks where possible as they can become a bottleneck. If there is any way you can work without them - do so.
2. Keep transactions short when you are issuing some changes that grab locks. The longer they stay open, the greater the chance for contention with another request.