Neo4j - Unable to rollback transaction in creating relationships with java

Question

I am trying to create some nodes in Neo4j through a Maven Java Application and create relationships between those nodes. Exactly i want to create 16807 nodes and 17210368 relationships. I read a file and get the row variable which has the number of nodes i must create and i also have a list which has 34420736 elements (=17210368*2). I want to create a relationship from node[element 0 of list] to node[element 1 from list], from node[element 2 of list] to node[element 3 from list] etc. Also the max element of list is 16807. I create an ArrayList<Node> as to create the nodes dynamically cause i want the programm to run with different files(and with different row values).

Here is my code:

    GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
    GraphDatabaseService graphDb = dbFactory.newEmbeddedDatabase("C:\Users\....\default.graphdb");

    Transaction tx = graphDb.beginTx();
    try {
        final RelationshipType type2 = DynamicRelationshipType.withName("KNOW");

        ArrayList<Node> nodelist = new ArrayList<Node>();

        for (int k = 0; k < row; k++) { //row=16807
            nodelist.add(graphDb.createNode());
            nodelist.get(k).setProperty("Name", "ListNode  " + k);
        }       

        int count=0;
        for (int j = 0; j < list.size() ; j++) { //list.size()=34420736
            nodelist.get(list.get(count)).createRelationshipTo(nodelist.get(list.get(count+1)), type2);
            count=count+2;
        }

        tx.success();
    }
    finally {
        tx.close();
    }
    graphDb.shutdown();

If i run the code without trying to create relationships it creates the nodes and run correctly. When i add the for loop that creates the realtionships it throws me the following error:

Exception in thread "main" org.neo4j.graphdb.TransactionFailureException: Unable to rollback transaction
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:131)
at com.mycompany.traverse_test.traverse_main.main(traverse_main.java:138)
Caused by: java.lang.IllegalStateException: No RelationshipState for added relationship!
at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter$1.visit(RelationshipChangeVisitorAdapter.java:132)
at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAddedRelationship(RelationshipChangeVisitorAdapter.java:83)
at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:106)
at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:47)
at org.neo4j.kernel.impl.util.diffsets.DiffSets.accept(DiffSets.java:76)
at org.neo4j.kernel.impl.api.state.TxState.accept(TxState.java:156)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.rollback(KernelTransactionImplementation.java:542)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:404)
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:112)
... 1 more

Any ideas??

Answer 1

Neo4j is trying to roll back your transaction due to a bug in your code. The fact that it is failing to roll back may be a bug in neo4j, but that is really not your main problem.

Looking at your code, it looks like you are iterating through your list too many times. That is, the code in the list loop is using up 2 list elements at a time, so you should only be looping list.size()/2 times.

Here is code that should fix that bug, and it also makes a few other improvements.

GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
GraphDatabaseService graphDb = dbFactory.newEmbeddedDatabase("C:\Users\....\default.graphdb");

Transaction tx = graphDb.beginTx();
try {
    final RelationshipType type2 = DynamicRelationshipType.withName("KNOW");

    ArrayList<Node> nodelist = new ArrayList<Node>();

    for (int k = 0; k < row; k++) { //row=16807
        Node node = graphDb.createNode();
        node.setProperty("Name", "ListNode  " + k);
        nodelist.add(node);
    }       

    for (int j = 0; j < list.size() ; j += 2) { //list.size()=34420736
        nodelist.get(list.get(j)).createRelationshipTo(
            nodelist.get(list.get(j+1)), type2);
    }

    tx.success();
} catch(Throwable e) {
    e.printStackTrace();
    // You may want to re-throw the exception, rather than just eating it here...
} finally {
    tx.close();
}
graphDb.shutdown();

[EDITED]

However, the above code can still run out of memory, since it is trying to create so many resources (16K nodes and 17M relationships) in a single transaction.

The following example code does the work in multiple transactions (one for creating the nodes and node list, and multiple transactions for the relationships).

NUM_RELS_PER_CHUNK specifies the maximum number of relationships to be created in each transaction. The createRelEndpointList() method must be modified to fill in the list of relationship endpoint (node) indices (each index is the 0-origin position of a node in nodeList).

public class MyCode {

    private static final int NODE_COUNT = 16807;
    private static final int NUM_RELS_PER_CHUNK = 1000000;

    public static void main(String[] args) {
        doIt();
    }

    private static void doIt() {
        GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
        GraphDatabaseService graphDb = dbFactory.newEmbeddedDatabase(new File("C:\\Users\\....\\default.graphdb"));

        try {
            RelationshipType type = DynamicRelationshipType.withName("KNOW");

            List<Node> nodeList = createNodes(graphDb, NODE_COUNT);
            List<Integer> list = createRelEndpointList();

            final int numRels = list.size() / 2;
            final int numChunks = (numRels + NUM_RELS_PER_CHUNK - 1)/NUM_RELS_PER_CHUNK;

            int startRelIndex = 0, endRelIndexPlus1;
            for (int i = numChunks; --i >= 0 && startRelIndex < numRels; ) {
                endRelIndexPlus1 = (i > 0) ? startRelIndex + NUM_RELS_PER_CHUNK : numRels;
                createRelationships(graphDb, nodeList, list, startRelIndex, endRelIndexPlus1, type);
                startRelIndex = endRelIndexPlus1;
            }
        } finally {
            graphDb.shutdown();
        }
    }

    private static List<Node> createNodes(GraphDatabaseService graphDb, int rowCount) {
        ArrayList<Node> nodeList = new ArrayList<Node>(rowCount);
        Transaction tx = graphDb.beginTx();
        try {
            final StringBuilder sb = new StringBuilder("ListNode  ");
            final int initLength = sb.length();
            for (int k = 0; k < rowCount; k++) {
                Node node = graphDb.createNode();
                sb.setLength(initLength);
                sb.append(k);
                node.setProperty("Name", sb.toString());
                nodeList.add(node);
            }
            tx.success();
            System.out.println("Created nodes.");
        } catch(Exception e) {
            e.printStackTrace();
            tx.failure();
            return null;
        } finally {
            tx.close();
        }
        return nodeList;
    }

    private static List<Integer> createRelEndpointList() {
        final List<Integer> list = new ArrayList<Integer>();

        // Fill
        //   list
        //     ...

        return list;
    }

    private static void createRelationships(GraphDatabaseService graphDb, List<Node> nodeList, List<Integer> list, int startRelIndex, int endRelIndexPlus1, RelationshipType type) {
        Transaction tx = graphDb.beginTx();
        try {
            final int endPlus2 = endRelIndexPlus1 * 2;
            for (int j = startRelIndex * 2; j < endPlus2; ) {
                Node from = nodeList.get(list.get(j++));
                Node to =   nodeList.get(list.get(j++));
                from.createRelationshipTo(to, type);
            }
            tx.success();
            System.out.println("Created rels. Start: " + startRelIndex + ", count: " + (endRelIndexPlus1 - startRelIndex));
        } catch(Exception e) {
            e.printStackTrace();
            tx.failure();
            // You may want to re-throw the exception, rather than just eating it here...
        } finally {
            tx.close();
        }
    }

}