CODEWITHSUNDEEP
node.jsmongodb
vashishatashu
vashishatashu

Reputation: 8142

Slow query blocks fast queries until slow query gets executed : MongoDB with NodeJs driver

I'm running a slow query/aggregate taking 3+ seconds, then other query get blocked until the slow query completes.

After a slow query is executed, only a number of fast queries execute that is equal to connection pool size and then all further operations get blocked until slow query executes. After that fast queries execute normally.

I'm using MongoDB 2.6.7 and mongodb NodeJs driver 1.4.30.

Slow aggregate pipeline :

[{"$unwind": "$status_history"},{"$sort": {"_id": -1}},{"$limit": 100}]

I'm running above query on a collection having 10k documents which on unwind results in 200k documents and then $sort operates. This takes about 5-10 seconds.

After this any simple queries that usually execute in 100-500ms 3-10 seconds.

Upvotes: 1

Views: 1578

Answers (2)

vashishatashu
vashishatashu

Reputation: 8142

This is what i got from MongoBD support :

poolSize : allows you to control how many tcp connections are opened in parallel. The default value for this is 5 but you can set it as high as you want. The driver will use a round-robin strategy to dispatch and read from the tcp connection.

If the connection is busy running a slow operation, this could block a subsequent operation.

There are two possible work-arounds:

Increase the connection pool size

Using a larger connection pool when running longer ops would reduce the likelihood/frequency of this happening if you have no control over the types of queries being run, though still possible that an operation would be blocked and that possibility would increase as more long running operations come in. With a bigger pool to round-robin through, it would be more likely that a long running op has completed on a connection by the time it is selected from the pool again.

Diverge the possible slower operations into another connection pool

You can create a separate connection pool (that is, result from MongoClient.connect() ) for the slower operations and leave the production pool to satisfy the fast queries. This would ensure that slow/blocking aggregation queries would not freeze the pools for other operations. Slower operations are of course hard to determine before hand, but if for example they could be queries using sort, a large limit, a skip, or an aggregation or map reduce operation.

Upvotes: 4

shahmanthan9
shahmanthan9

Reputation: 523

You need to optimize your queries using proper indexes because in MongoDB also there is locking of db, so all queries will be blocked for the sometime. Please check here

Breaking down your big scan in multiple small scans and joining result sets also helps in most cases.

Also if possible create sharding with replica sets so you can distribute your queries

Upvotes: 0

Related Questions