CODEWITHSUNDEEP
mongodbamazon-ec2
Jim Jose
Jim Jose

Reputation: 1319

MongoDB random slow queries - EC2 IOPS

I have a mongodb(version: 2.0.8) installation on an EC2 instance (7GB RAM) and the data size is still less than 1GB. I am using provisioned 100 IOPS disk for increased disk performance.

The problem is, I am getting a number of random SLOW QUERIES like the one below, (from mongo log)

db.UserInfoShared query: { _id: "999081873179" } ntoreturn:1 idhack:1 reslen:108 1919ms

Took almost 2 secs !

It is just an _id lookup on a collection with around 100,000 entries each less than 500 bytes in size. The instance has only mongo running in it and normally such lookups takes less than 0.01 sec.

What could the possible cause for this? what should I try to get it resolved? Any help is much appreciated...

Upvotes: 2

Views: 1968

Answers (1)

Jim Jose
Jim Jose

Reputation: 1319

@angela, @Asya, thanks a lot for helping me out.

As it turned out the main cause of latency was mongodb GLOBAL LOCK itself. We had a lock percentage which was averaging at 5% and sometime peaks up to 30-50% and that results in the slow queries.

If you are facing this issue, the first thing you have to do it enable mongodb MMS service (mms.10gen.com), which will give you a lot of insights on what exactly is happening in your db server.

In our case the LOCK PERCENTAGE was really high and there were multiple reasons for it. First thing you have to do to figure it out is to read mongodb documentation on concurrency, http://docs.mongodb.org/manual/faq/concurrency/

The reason for lock can be in application level, mongodb or hardware.

1) Our app was doing a lot of updates and each update(more than 100 ops/sec) holds a global lock in mongodb. The issue was that when an update happens for an entry which is not in memory, mongo will have to load the data into memory first and then update (in memory) and the whole process happens while inside the global lock. If say the whole thing takes 1 sec to complete (0.75sec to load the data from disk and 0.25sec to update in memory), the whole rest of the update calls waits (for the entire 1 sec) and such updates starts queuing up... and you will notice more and more slows requests in your app server.

The solution for it (while it might sound silly) is to query for the same data before you make an update. What it effectively does is moving the 'load data to memory' (0.75sec) part out of the global lock which greatly reduces your lock percentage

2) The other main cause of global lock is mongodb's data flush to disk. Basically in every 60sec (or less) mongodb (or the OS) writes the data to disk and a global lock is held during this process. (This kinda explains the random slow queries). In your MMS stats, see the graph for background flush avg... if its high, that means you have to get faster disks.

In our case, we moved to a new EBS optimized instance in EC2 and also bumped our provisioned IOPS from 100 to 500 which almost halved the background flush avg and the servers are much happier now.

Upvotes: 3

Related Questions