CODEWITHSUNDEEP
c++mongodbindexing
TimoK
TimoK

Reputation: 253

C++ MongoClient index optimization for mass data bulk inserts

I am developing an application which inserts data into MongoDB in high frequency (thousands of documents sub-second). As such, index and storage space optimization is key.

So, before inserting the first record (collection names are dynamic), I would like to do the following with the C++ driver:

My insert command is then conn.insert(coll, vec); which obviously works fine for any number of vector elements.

Help is greatly appreciated!

Upvotes: 2

Views: 555

Answers (1)

WiredPrairie
WiredPrairie

Reputation: 59763

I'm not sure I understand why you're removing the _id index and replacing it with another index, yet still setting the _id field.

  1. Apparently, you can disable the _id for a collection if needed, by extending the method createCollection from the DbClientWithCommands (documentation) class. Of course, you need to also make sure that an _id is not automatically inserted by the driver automatically (many drivers to this, so for some people, this will still be an issue).
  2. The current driver method ensureIndex has a background parameter you can provide (documentation)
  3. I'm not aware of any way to programmatically control the padding. It's automatically determined by MongoDB over time for a collection. If you aren't modifying documents, I'd expect it to near be near 1 (meaning there's no padding). Check the stats to be sure.

For creating a collection without an _id and using autoIndexId, you'd need to create a new function, as the built-in one currently does, you'd need to copy the code as mentioned above and do this:

bool MyClass::createCollection(const string &ns, long long size, 
                               bool capped, int max, bool disableAutoIndexId, BSONObj *info) {
    verify(!capped||size);
    BSONObj o;
    if ( info == 0 )    info = &o;
    BSONObjBuilder b;
    string db = nsToDatabase(ns);
    b.append("create", ns.c_str() + db.length() + 1);
    if ( size ) b.append("size", size);
    if ( capped ) b.append("capped", true);
    if ( max ) b.append("max", max);
    if ( disableAutoIndexId ) b.append("autoIndexId", false);
    return runCommand(db.c_str(), b.done(), *info);
}

Upvotes: 1

Related Questions