CODEWITHSUNDEEP
mongodbindexingmongodb-query
IamIC
IamIC

Reputation: 18259

MongoDB limit to number of sparse indexes

I have a specific need to have a particularly large number of sparse indexes in a MongoDB collection. The number would likely be in the low thousands. The documents stored would each use only a relatively small number of indexes, around 10 - 30 each on average. However, the documents are highly varied, and will require different sets of indexes each.

I cannot find any documentation as to the likely effects of having > 1000 sparse indexes on a collection. Space is not a concern. My specific concern is the impact on save time and query set-up. Will MongoDB get bogged down with this large overhead?

I will run my own benchmarks, but I'd like to know a) if someone knows of some official guidelines on this and b) has any experience with this scenario.

Upvotes: 1

Views: 440

Answers (2)

Nic Cottrell
Nic Cottrell

Reputation: 9685

Until this is implemented in the server, you could run your OR query twice - once for each field and merge the results in the client.

For example if you had objects:

{_id: 123, foo: 12}
{_id: 124, bar: 13}
{_id: 124, foo: 12, bar: 15}

You could have one sparse index {foo:1, _id:1} and another {bar:1, _id:1} and do queries like find({foo:12}, {_id:1}) and find({bar:13}, {_id:1}) and then OR or AND the object Ids in the client. You could then retrieve the full objects for just those matching IDs.

Note: by retrieving just the _id field in the find queries, Mongo can return the results from the index and doesn't need to unpack any BSON to give the results making this very quick.

Upvotes: 1

Eve Freeman
Eve Freeman

Reputation: 33175

There is a namespace limitation (total number of indexes, collections, etc.) that you might run into, but that can be lifted with --nssize: http://www.mongodb.org/display/DOCS/Using+a+Large+Number+of+Collections

The inserts will have some overhead added, but it should "fail fast" once it realizes the document you're inserting doesn't need to be added to most of the indexes. Disclaimer: I haven't tried benchmarking nearly as many as you're considering. I'm interested to see if it will work.

One thing to keep in mind is that you'll not be able to use more than one index per query without using $or, which is a current limitation in mongodb. http://www.mongodb.org/display/DOCS/Indexing+Advice+and+FAQ#IndexingAdviceandFAQ-Oneindexperquery.

Upvotes: 1

Related Questions