Mongo running slow on Amazon EC2

Question

I have been using compose.io to host my Mongo instance. My data is getting bigger and its becoming cost prohibitive to stay there so I'd like to move to EC2 where I have ~$750 in credits.

The problem:

I have an endpoint to authenticate a user that I run from my localhost:

When my API is pointed at the compose.io database its ~200ms response time

When my API is pointed at my new EC2 Mongo instances its ~700ms response time.

(The databases are exact copies)

Pinging the EC2 instance is ~90-100ms.

The collections have been reIndexed() and there is zero load on the Mongo instance.

Details on the EC2 instance:

M3.large

100 Provision iOPs

Zero traffic /load.

I cannot figure out why Mongo is just so slow to respond. When I authenticate several things occur against the database (Here is the output from mongod.log)

2015-11-22T13:44:15.631+0000 [conn6] insert production.transactions query: { method: "POST", resource: "/api/v1.1/login", body: { password: "***********", email: "********" }, timezone: "America/New_York", agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36", _id: ObjectId('5651c6afdc24948b9f2e15e5'), created: new Date(1448199855577), __v: 0 } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:99 0ms
2015-11-22T13:44:15.631+0000 [conn6] command production.$cmd command: insert { insert: "transactions", documents: [ { method: "POST", resource: "/api/v1.1/login", body: { password: "***********", email: "*******" }, timezone: "America/New_York", agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36", _id: ObjectId('5651c6afdc24948b9f2e15e5'), created: new Date(1448199855577), __v: 0 } ], ordered: false, writeConcern: { w: 1 } } keyUpdates:0 numYields:0 locks(micros) w:83 reslen:40 0ms
2015-11-22T13:44:15.734+0000 [conn8] query production.users query: { email: "*******" } planSummary: IXSCAN { email: 1 } ntoskip:0 keyUpdates:0 numYields:0 locks(micros) r:125 nreturned:1 reslen:1553 0ms
2015-11-22T13:44:15.918+0000 [conn7] remove production.tokens query: { user_id: "5330d44ba6885a020005bc88" } ndeleted:0 keyUpdates:0 numYields:0 locks(micros) w:236 0ms
2015-11-22T13:44:15.918+0000 [conn7] command production.$cmd command: delete { delete: "tokens", deletes: [ { q: { user_id: "5330d44ba6885a020005bc88" }, limit: 0 } ], ordered: true, writeConcern: { w: 1 } } keyUpdates:0 numYields:0  reslen:40 0ms
2015-11-22T13:44:16.019+0000 [conn9] insert production.tokens query: { _id: "Tor9ke2lrt5Ooeifuh6hnCYFmmpDlWu8tRu2T2uZbgylFpx8EBlg1Aw7cQKQNc0I09zRhLrxdceV7lTf6UWl769ZMLX1cxlb0qksY8ssj1zme9uT1PkpNlIlNdBJE40S", user_id: "5330d44ba6885a020005bc88", expires: new Date(1448804120000), created: new Date(1448199855964), __v: 0 } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:73 0ms
2015-11-22T13:44:16.020+0000 [conn9] command production.$cmd command: insert { insert: "tokens", documents: [ { _id: "Tor9ke2lrt5Ooeifuh6hnCYFmmpDlWu8tRu2T2uZbgylFpx8EBlg1Aw7cQKQNc0I09zRhLrxdceV7lTf6UWl769ZMLX1cxlb0qksY8ssj1zme9uT1PkpNlIlNdBJE40S", user_id: "5330d44ba6885a020005bc88", expires: new Date(1448804120000), created: new Date(1448199855964), __v: 0 } ], ordered: false, writeConcern: { w: 1 } } keyUpdates:0 numYields:0 locks(micros) w:105 reslen:40 0ms
2015-11-22T13:44:16.128+0000 [conn10] command production.$cmd command: findAndModify { findandmodify: "users", query: { _id: ObjectId('5330d44ba6885a020005bc88') }, new: false, remove: false, upsert: false, update: { $set: { last_login: new Date(1448199856068) } }, writeConcern: { w: 1 } } update: { $set: { last_login: new Date(1448199856068) } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:1 fastmod:1 keyUpdates:0 numYields:0 locks(micros) w:130 reslen:1624 0ms

Answer 1

The cause of the issue appears to be setting up a mongo instance in US west while hosting the application in US East. Network latency is the cause of the increased response time.