Neo4j Embedded Java - Transaction was marked as successful, but unable to commit transaction so rolled back

Question

I am trying to load 16807 nodes with 17.210.368 relationships in a graph in Neo4j. As to do so, i load a file that contains a vincinity table and i get a list with the nodes that must be connected with a relationship.

Find my code below:

    String inputFile = "Desktop\kron7cd_unix.t01";
    FileInputStream in = new FileInputStream(inputFile);
    FileChannel ch = in.getChannel();
    ByteBuffer buf = ByteBuffer.allocate(1024);

    ArrayList<Integer> list = new ArrayList<Integer>();
    int NumOfOnes = 0;
    int column=-1;
    int row=0;
    int rd;
    while ((rd = ch.read( buf )) != -1){
        buf.flip();
        while (buf.hasRemaining()){
            byte byteVal = buf.get();
            if((byteVal == 48) || (byteVal == 49)){// when finds 1 or 0
                column++;
            }
            if (byteVal == 92){//when finds '/'
                    row++;
                    column=-1;
            }
            if(byteVal == 49){//when finds 1
                NumOfOnes++;
                list.add(column);
                list.add(row);
            }
        }
    buf.clear();
    }
    ch.close();


    GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
    GraphDatabaseService graphDb = dbFactory.newEmbeddedDatabase("C:\Neo4j\default.graphdb");

    Transaction tx = graphDb.beginTx();
    try {

        Label myLabel = DynamicLabel.label("Data");
        ArrayList<Node> nodelist = new ArrayList<Node>();

        for (int k = 0; k < row; k++) {
            nodelist.add(graphDb.createNode());
        }

        for (int k = 0; k < row; k++) {
            nodelist.get(k).setProperty("ID", k);
            nodelist.get(k).setProperty("Group","Random");
            nodelist.get(k).addLabel(myLabel);
        }

        Relationship rel;
        final RelationshipType type2 = DynamicRelationshipType.withName("Rel");

        for (int j = 0; j < list.size()-1 ; j += 2) { //list.size()=34420736

            rel = nodelist.get(list.get(j)).createRelationshipTo(nodelist.get(list.get(j+1)), type2);
            rel.setProperty("Team", "Common");

            if (j > 0 && j % 10000 == 0) {// as to commit transaction every now and then and dont throw heap space
                tx.success();
                tx.close();
                tx = graphDb.beginTx();
            }                

        }

        tx.success();
    }

    finally {
        tx.close();
    }
    graphDb.shutdown();

When i run this code it throws me the following error. I use Neo4j 2.3.3 and Netbeans 8.1 with Java 8. I want to understand if the problem is the heap space or when it tries to commit the transaction. I have also add the command-line option -Xmx1g in my project as to increrase the heap space.

Any ideas?

Find error messages below:

Exception in thread "main" org.neo4j.graphdb.TransactionFailureException: Transaction was marked as successful, but unable to commit transaction so rolled back.
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:121)
at com.mycompany.traverse_test.traverse_main.main(traverse_main.java:232)

Caused by: org.neo4j.kernel.api.exceptions.TransactionFailureException: Could not apply the transaction to the store after written to log
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:105)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.commit(TransactionRepresentationCommitProcess.java:58)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit(KernelTransactionImplementation.java:565)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:458)
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:97)
... 1 more

Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:386)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager$Slab.<init>(MemoryManager.java:111)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager.allocateAligned(MemoryManager.java:82)
at org.neo4j.io.pagecache.impl.muninn.MuninnPage.initBuffer(MuninnPage.java:417)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:230)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:157)
at org.neo4j.io.pagecache.impl.muninn.MuninnWritePageCursor.next(MuninnWritePageCursor.java:58)
at org.neo4j.kernel.impl.store.PropertyStore.updateRecord(PropertyStore.java:144)
at org.neo4j.kernel.impl.transaction.command.NeoStoreTransactionApplier.visitPropertyCommand(NeoStoreTransactionApplier.java:99)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visitPropertyCommand(CommandApplierFacade.java:120)
at org.neo4j.kernel.impl.transaction.command.Command$PropertyCommand.handle(Command.java:288)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:82)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:45)
at org.neo4j.kernel.impl.transaction.log.PhysicalTransactionRepresentation.accept(PhysicalTransactionRepresentation.java:69)
at org.neo4j.kernel.impl.api.TransactionRepresentationStoreApplier.apply(TransactionRepresentationStoreApplier.java:111)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:100)
... 5 more

Answer 1

As i realize by trying to run the same code in another system i understand that the problem was the heap space error. I was running the code in my system with 3GB RAM and it pops the above error, but wen i run it in a system with 12GB RAM it runs normally. (i suppose with even 8GB RAM it will not have a problem)

Answer 2

You should commit the transactions every 1k - 10k elements to avoid holding all the data on the heap. See, for example, https://github.com/graphaware/neo4j-framework/tree/master/tx-executor#batch-transactional-operations