CODEWITHSUNDEEP
javaneo4j
Abhinav Upadhyay
Abhinav Upadhyay

Reputation: 2585

Failing to query Neo4J database with Out of memory errors

We have a moderate size graph having 6330318 nodes and 6779549 relationships. After having imported the data, I have been unable to start the neo4j server or the neo4j shell. Both of them fail with GC overhead limit exceeded errors. I am giving a maximum heap space size of 12 GB to the JVM. The server has 15GB RAM. The neo4j.properies has following configuration for memory mapped IO. 

neostore.nodestore.db.mapped_memory=400M
neostore.relationshipstore.db.mapped_memory=520M
neostore.propertystore.db.mapped_memory=100M
neostore.propertystore.db.strings.mapped_memory=10M
neostore.propertystore.db.arrays.mapped_memory=10M

We don't have any properties in the graph, therefore such low values for propertystore related settings.

I have also written a small web application for querying the graph, which is deployed on Tomcat. That also failed to query the graph with the following errors

java.lang.OutOfMemoryError: Java heap space
    java.util.Arrays.copyOf(Arrays.java:2245)
    java.util.Arrays.copyOf(Arrays.java:2219)
    java.util.ArrayList.grow(ArrayList.java:242)
    java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216)
    java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208)
    java.util.ArrayList.add(ArrayList.java:440)
    org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:216)
    org.apache.lucene.index.FieldInfos.read(FieldInfos.java:361)
    org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74)
    org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:80)
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
    org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
    org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
    org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
    org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
    org.apache.lucene.index.IndexReader.open(IndexReader.java:256)
    org.neo4j.kernel.api.impl.index.LuceneLabelScanStore.init(LuceneLabelScanStore.java:249)
    org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:483)
    org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:72)
    org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.start(NeoStoreXaDataSource.java:423)
    org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507)
    org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
    org.neo4j.kernel.impl.transaction.XaDataSourceManager.start(XaDataSourceManager.java:164)
    org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507)
    org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
    org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:339)
    org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:59)
    org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:90)
    org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:199)
    com.socialtwist.rnet.web.RnetNeo4jServlet.init(RnetNeo4jServlet.java:44)

We have deployed the application on a server with 16 GB of RAM and have given Tomcat a heap space of 12 GB. What other tuning options are available for Neo4J to make this work under reasonable response time?

Upvotes: 1

Views: 322

Answers (1)

Michael Hunger
Michael Hunger

Reputation: 41706

How many labels do you have in your graph? Something is very off with your graph and query. I think you confused labels (types) with id's

Create index on :User(id);

MATCH (referee:User)<-[:IS_FRIEND_OF]-(referrer:User) WHERE referee.id=1677523 RETURN referrer;

Should return instantly after you fixed you graph model, i.e. Removed all numeric labels

Upvotes: 2

Related Questions