CODEWITHSUNDEEP
neo4j
Silfverstrom
Silfverstrom

Reputation: 29338

Neo4j Batch insertion - Index causes OutOfMemoryError exception

I'm adding a large dataset to a neo4j server, using a "batchimport"

BatchInserterIndex someIndex = indexProvider.nodeIndex("some_index", MapUtil.stringMap( "type", "exact") );

    /** Looping large dataset */
    for (..) {

        Map<String, Object> testMap //Map with various properties    
        long id  = inserter.createNode(testMap);
        someIndex.add(id, testMap);
    }

After approximately 400k entries, my code throws an exception "java.lang.OutOfMemoryError: Java heap space".

It works perfectly when I'm not using a index. I've tried flushing the index from time to time, but no improvement.

How can I add a large dataset while at the same time using a index? Any idea where I went wrong?

Upvotes: 1

Views: 231

Answers (1)

Mattias Finn&#233;
Mattias Finn&#233;

Reputation: 3054

If your heap isn't all that big you could run into an issue where the batch commit size on the lucene index is too high. The default is 500,000 values. And I'm afraid you cannot change it from the public API. I'll try to get something in that could configure this batch size for version 1.9.

Upvotes: 1

Related Questions