High response time for neo4j queries

Question

I am having a neo4j system (2.0.1 version) having more than 2 million Nodes. Its a social network with different types of entities and nodes , having high cardinality nodes (2000 likes and 500 followers for power users).

I am running the system on an Amazon AWS EC2 c3.2xlarge, 8 cores 16GB RAM . I have a mongodb Instance running on the same server.

But once in 2-3 days without any noticeable jump in CPU usage, response time shoot up, simple indexed queries like find by index takes 10s instead of ~100ms.

I am not able to pinpoint the exact problem. I am providing Neo4j's messages.log file, please check if any obvious mistakes present.

messages.log file

UPDATE: JVM Settings are as follows:

wrapper.java.additional=-XX:+UseConcMarkSweepGC 
wrapper.java.additional=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional=-XX:NewRatio=1
wrapper.java.additional=-XX:SurvivorRatio=32
wrapper.java.additional=-XX:MaxPermSize=100m
wrapper.java.additional=-d64
wrapper.java.additional=-server
wrapper.java.additional=-Xss4096k


wrapper.java.additional=-Xloggc:data/log/neo4j-gc.log
wrapper.java.additional=-XX:+PrintGCDetails
wrapper.java.additional=-XX:+PrintGCDateStamps
wrapper.java.additional=-XX:+PrintGCApplicationStoppedTime


# Initial Java Heap Size (in MB)    
wrapper.java.initmemory=10240


# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=10240

UPDATE 2 : My Neostore files :

$ ls -lht *store*

-rw-rw-r-- 1 ubuntu ubuntu   63 Sep 20 11:50 neostore
-rw-rw-r-- 1 ubuntu ubuntu 576M Sep 20 11:50 neostore.propertystore.db
-rw-rw-r-- 1 ubuntu ubuntu 282M Sep 20 11:50 neostore.relationshipstore.db
-rw-rw-r-- 1 ubuntu ubuntu 1.1G Sep 20 11:50 neostore.propertystore.db.strings
-rw-rw-r-- 1 ubuntu ubuntu  32M Sep 20 11:50 neostore.nodestore.db
-rw-rw-r-- 1 ubuntu ubuntu 549K Sep 20 11:11 neostore.relationshipstore.db.id
-rw-rw-r-- 1 ubuntu ubuntu 3.9K Sep 18 16:35 neostore.propertystore.db.index.keys
-rw-rw-r-- 1 ubuntu ubuntu  927 Sep 18 16:35 neostore.propertystore.db.index
-rw-rw-r-- 1 ubuntu ubuntu    9 Sep 18 11:21 neostore.id

Memory map settings are as follows:

neostore.nodestore.db.mapped_memory=100M
neostore.relationshipstore.db.mapped_memory=400M
neostore.propertystore.db.mapped_memory=800M
neostore.propertystore.db.strings.mapped_memory=1536M

Answer 1

It is hard to put this into an answer as there will be a lot of variables, but it might well be that your heap is too big. I see that you have GC logging setup, have you looked at the file data/log/neo4j-gc.log.

Importantly from your messages.log you can see things slowing down with so many Garbage Collection events, a particularly bad patch being over a period of under 2 minutes with a time blocked by GC around 80 seconds (I think you stopped the server shortly after this).

2014-09-18 05:45:09.323+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 21640ms [total block time: 5232.321s]
2014-09-18 05:45:33.694+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 19016ms [total block time: 5251.337s]
2014-09-18 05:45:57.579+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 18985ms [total block time: 5270.322s]
2014-09-18 05:46:22.333+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 19929ms [total block time: 5290.251s]

You have provided the JVM settings, it would be helpful to also have an idea of how you have configured Neo in terms of memory maps? Also, the size of your neostore files on disk (just look in your graph directory).

You have set the JVM to use up to 10Gb of RAM, leaving you with 6Gb for the OS, Memory mapping and Mongo, is that enough?

A quick look at the recommended settings in the Neo documentation would suggest that at 2 Million Nodes you would want 512Mb of Heap. Obviously that is a rule of thumb and the amount of data you store on each node and the amount of relationship data will clearly impact this.

Using the Neo4J calculator I plugged in 2000000 nodes, 10000000 relationships, 100 Bytes of data per relationship and it suggested a machine with 12Gb RAM running a JVM with a 6Gb heap. Reading the memory mapping configuration I think you would then want to try mapping about 3.5Gb to your graph data sensibly split for your dataset. Using there example:

neostore.nodestore.db.mapped_memory=45M
neostore.relationshipstore.db.mapped_memory=3G
neostore.propertystore.db.mapped_memory=50M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=0M

That is based on their rough calculations of:

number_of_nodes * 9 bytes
number_of_relationships * 33 bytes

There is obviously a trade off between on heap storage, memory maps and then disk caches and disk access, each getting at least an order of magnitude slower (I think), but garbage collection is expensive, especially when the heap is quite large.