Reputation: 83
I have a bit of confusion regarding memory management.
I am using neo4j in an embedded mode within a java application. Version - 1.8.1. JVM version - 1.6
I have a single threaded loader with a for loop that contains a nested for loop. Inside the external for loop, I do graphDb.beginTx();
, get an id of a test object from mysql and create a Node for that. In the inner for loop, I am querying mysql to get a bunch of related objects. I create nodes for them and the corresponding relationship with the node that I created in the external loop.
At the end of the external for loop, I do tx.success()
and tx.finish()
.
for (int x = 0; x < 10000000; x++) {
Transaction tx = graphDb.beginTx();
Node n = graphDb.createNode();
Long id = n.getId();
System.out.println("ID: " + id);
n.setProperty("TestId", x);
* * * get the mysql ids * * * *
for (int y = 0; y < mysqlidlist; y++) {
Node n1 = graphDb.createNode();
Long id2 = n1.getId();
n1.setProperty("InnerTestId", y);
Relationship rel = n.createRelationshipTo(n1, NodeRelation.ATTRIBUTE);
rel.setProperty("Weight", 0.3);
}
tx.success();
tx.finish();
tx = null;
}
Based on my readings, I thought that Neo4j would then release the memory that the transaction had grabbed. However, I see the memory usage always increasing and after some time it hits the Xmx setting. I will profile it to make sure there are no other leaks. I am setting all the other variables to null at the end and that should help GC reap it in a better way.
Am I wrong in my understanding? If so, what is the recommended best practice on memory management?
Next step - the Single Threaded loader will be extended to be a multi-threaded one and as such, want to make sure that the memory management and transaction management is robust.
Thanks so much!
Regards,
Sachin
Upvotes: 3
Views: 987
Reputation: 497
For this kind of massive insertion if you don't need transactions I suggest to use Batch Insertion.
http://docs.neo4j.org/chunked/milestone/batchinsert.html
Or, not recommended: If you want to use your snippet of code add something like: if(y%1000==0) { tx.success();tx.finish()} inside the for loop to commit and release the memory used every n inserts, but it will be slower
Upvotes: 1
Reputation: 33185
You could try turning off caching (use cache_type=none). docs.neo4j.org/chunked/milestone/configuration-caches.html
Upvotes: 0