CODEWITHSUNDEEP
mongodb
Kos
Kos

Reputation: 3

MongoDB can not find() in 1 million documents

I just started to deal with MongoDB. Created 10 thousand json documents. I do search:

db.mycollection.find({"somenode1.somenode2.somenode3.somenode4.Value", "9999"}).count()

It gives out the correct result. Operating time: 34 ms. Everything is OK.

Now create a database with 1 million of the same documents. The total size of the database is 34Gb.The MongoDB divided the database into files by 2Gb. I repeat the above described query to find the number of relevant documents. I waited for result about 2 hours. The memory was occupied (16GB). Finally I shut down the Mongo. System: Windows 7 x64, 16Gb RAM.

Please tell me what I'm doing wrong. A production db will be much bigger.

Upvotes: 0

Views: 1554

Answers (1)

SDillon
SDillon

Reputation: 313

In your particular case, it appears you simply do not have enough RAM. At minimum, and index on "somenode4" would improve the query performance. Keep in mind, the indexes are going to want to be in RAM as well so you may need more RAM anyhow. Are you on a virtual machine? If so; I recommend you increase the size of the machine to account for the size of the working set.

As one of the other commenters stated, that nesting is a bit ugly but I understand it is what you were dealt. So other than RAM, indexing appears to be your best bet.

As part of your indexing effort, you may also want to try experimenting with pre-heating the indexes to ensure they are in RAM prior to that find and count(). Try executing a query that seeks for something that does not exist. This should force the indexes and data into RAM prior to that query. Pending how often your data changes, you may want this to be done once a day or more. You are essentially front-loading the slow operations.

Upvotes: 1

Related Questions