Neo4j Java API Concurrency v2.0M3: Exception when iterating over relationships while other threads creating new relationships concurrently

Question

What I try to achieve here is to get the number of relationships of a particular node, while other threads adding new relationships to it concurrently. I run my code in a unit test with TestGraphDatabaseFactory().newImpermanentDatabase() graph service.

My code is executed by ~50 threads, and it looks something like this:

int numOfRels = 0;
try {
    Iterable<Relationship> rels = parentNode.getRelationships(RelTypes.RUNS, Direction.OUTGOING);
    while (rels.iterator().hasNext()) {
        numOfRels++;
        rels.iterator().next();
    }
}
catch(Exception e) {
    throw e;
}

// Enforce relationship limit
if (numOfRels > 10) {
    // do something
}

Transaction tx = graph.beginTx();
try {
    Node node = createMyNodeAndConnectToParentNode(...);

    tx.success();

    return node;
}
catch (Exception e) {
    tx.failure();
}
finally {
    tx.finish();
}

The problem is once a while I get a "ArrayIndexOutOfBoundsException: 1" in the try-catch block above (the one surrounding the getRelationships()). If I understand correctly Iterable is not thread-safe and causing this problem.

My question is what is the best way to iterate over constantly changing relationships and nodes using Neo4j's Java API?

I am getting the following errors:

Exception in thread "Thread-14" org.neo4j.helpers.ThisShouldNotHappenError: Developer: Stefan/Jake claims that: A property key id disappeared under our feet
    at org.neo4j.kernel.impl.core.NodeProxy.setProperty(NodeProxy.java:188)
    at com.inbiza.connio.neo4j.server.extensions.graph.AppEntity.createMyNodeAndConnectToParentNode(AppEntity.java:546)
    at com.inbiza.connio.neo4j.server.extensions.graph.AppEntity.create(AppEntity.java:305)
    at com.inbiza.connio.neo4j.server.extensions.TestEmbeddedConnioGraph$appCreatorThread.run(TestEmbeddedConnioGraph.java:61)
    at java.lang.Thread.run(Thread.java:722)
Exception in thread "Thread-92" java.lang.ArrayIndexOutOfBoundsException: 1
    at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:72)
    at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
    at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
    at com.inbiza.connio.neo4j.server.extensions.graph.AppEntity.create(AppEntity.java:243)
    at com.inbiza.connio.neo4j.server.extensions.TestEmbeddedConnioGraph$appCreatorThread.run(TestEmbeddedConnioGraph.java:61)
    at java.lang.Thread.run(Thread.java:722)
Exception in thread "Thread-12" java.lang.ArrayIndexOutOfBoundsException: 1
    at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:72)
    at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
    at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
    at com.inbiza.connio.neo4j.server.extensions.graph.AppEntity.create(AppEntity.java:243)
    at com.inbiza.connio.neo4j.server.extensions.TestEmbeddedConnioGraph$appCreatorThread.run(TestEmbeddedConnioGraph.java:61)
    at java.lang.Thread.run(Thread.java:722)
Exception in thread "Thread-93" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "Thread-90" java.lang.ArrayIndexOutOfBoundsException

Below is the method responsible of node creation:

static Node createMyNodeAndConnectToParentNode(GraphDatabaseService graph, final Node ownerAccountNode, final String suggestedName, Map properties) {

  final String accountId = checkNotNull((String)ownerAccountNode.getProperty("account_id"));

  Node appNode = graph.createNode();
  appNode.setProperty("urn_name", App.composeUrnName(accountId, suggestedName.toLowerCase().trim()));

  int nextId = nodeId.addAndGet(1); // I normally use getOrCreate idiom but to simplify I replaced it with an atomic int - that would do for testing 

  String urn = App.composeUrnUid(accountId,  nextId);
  appNode.setProperty("urn_uid", urn);
  appNode.setProperty("id", nextId);
  appNode.setProperty("name", suggestedName);

  Index<Node> indexUid =  graph.index().forNodes("EntityUrnUid");
  indexUid.add(appNode, "urn_uid", urn);

  appNode.addLabel(LabelTypes.App);

  appNode.setProperty("version", properties.get("version"));
  appNode.setProperty("description", properties.get("description"));

  Relationship rel = ownerAccountNode.createRelationshipTo(appNode, RelTypes.RUNS);
  rel.setProperty("date_created", fmt.print(new DateTime()));

  return appNode;
}

I am looking at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull()

It looks like my test generates a condition where else if ( (status = fromNode.getMoreRelationships( nodeManager )).loaded() || lastTimeILookedThereWasMoreToLoad ) is not executed, and where currentTypeIterator state is changed in between.

RelIdIterator currentTypeIterator = rels[currentTypeIndex];  //<-- this is where is crashes
do
{
  if ( currentTypeIterator.hasNext() )
  ...
  ... 

  while ( !currentTypeIterator.hasNext() )
  {
    if ( ++currentTypeIndex < rels.length )
    {
        currentTypeIterator = rels[currentTypeIndex];
    }
    else if ( (status = fromNode.getMoreRelationships( nodeManager )).loaded()
            // This is here to guard for that someone else might have loaded
            // stuff in this relationship chain (and exhausted it) while I
            // iterated over my batch of relationships. It will only happen
            // for nodes which have more than <grab size> relationships and
            // isn't fully loaded when starting iterating.
            || lastTimeILookedThereWasMoreToLoad )
    {
        ....
    }
  }
} while ( currentTypeIterator.hasNext() );

I also tested couple locking scenarios. The one below solves the issue. Not sure if I should use a lock every time I iterate over relationships based on this.

Transaction txRead = graph.beginTx();
try {
  txRead.acquireReadLock(parentNode);

  long numOfRels = 0L;
  Iterable<Relationship> rels = parentNode.getRelationships(RelTypes.RUNS, Direction.OUTGOING);
  while (rels.iterator().hasNext()) {
    numOfRels++;
    rels.iterator().next();
  }

  txRead.success();
}
finally {
  txRead.finish();
}

I am very new to Neo4j and its source base; just testing as a potential data store for our product. I will appreciate if someone knowing Neo4j inside & out explains what is going on here.

Answer 1

This is a bug. The fix is captured in this pull request: https://github.com/neo4j/neo4j/pull/1011

Answer 2

Well I think this a bug. The Iterable returned by getRelationships() are meant to be immutable. When this method is called, all the available Nodes till that moment will be available in the iterator. (You can verify this from org.neo4j.kernel.IntArrayIterator)

I tried replicating it by having 250 threads trying to insert a relationship from a node to some other node. And having a main thread looping over the iterator for the first node. On careful analysis, the iterator only contains the relationships added when getRelationship() was last called. The issue never came up for me.

Can you please put your complete code, IMO there might some silly error. The reason it cannot happen is that the write locks are in place when adding a relationship and reads are hence synchronized.