CODEWITHSUNDEEP
mongodb
user1275011
user1275011

Reputation: 1762

MongoDB Batch update performance problems

I understand that MongoDB supports batch inserts, but not batch updates.

Batch-inserting thousands of documents is pretty fast, but updating thousands of documents is incredibly slow. So slow that the workaround I'm using now is to remove the old documents and batch-insert the updated documents. I had to add some flags to mark them as invalid and all the stuff required for compensating from a failed mock 'bulk update'. :(

I know this is an awful and unsafe solution, but its the only way I've been able to reach the required performance.

If you know a better way, please help me.

Thanks!

Upvotes: 2

Views: 6910

Answers (2)

Thamisetty Krishna Mohan
Thamisetty Krishna Mohan

Reputation: 11

I had a similar situation, after doing trial and error, I created an index in MongoDB or through mongoose, now thousands of documents is pretty fast by using bulk operations i.e. bulk.find({}).upsert.update({}).

Example:

var bulk = items.collection.initializeOrderedBulkOp();
bulk.find({fieldname: value, active: false}).updateOne({.upsert().updateOne({
    $set: updatejsondata,
    $setOnInsert: createjsondata        
});
Note: you need to use $push for storing in array like $set, you need to include $push 
example:
bulk.find({name: value, active: false}).updateOne({.upsert().updateOne({
    $set: updatejsondata,
    $push: {logdata: filename + " - " + new Date()} 
    $setOnInsert: createjsondata        
});

Creating index: In above case on Items collection you need to create the index on search fields i.e. name, active

Example:
Mongo Command Line: 

db.items.ensureIndex({name: 1, active: 1}, {unique: false, dropDups: false})

Mongoose Schema: 

ItemSchema.index({name: 1, active: 1}, {name: "itemnameactive"});

Hope this will help you out doing bulk operations

Upvotes: 1

JohnnyHK
JohnnyHK

Reputation: 311835

As long as you're using MongoDB v2.6 or higher, you can use bulk operations to perform updates as well.

Example from the docs:

var bulk = db.items.initializeUnorderedBulkOp();
bulk.find( { status: "D" } ).update( { $set: { status: "I", points: "0" } } );
bulk.find( { item: null } ).update( { $set: { item: "TBD" } } );
bulk.execute();

Upvotes: 4

Related Questions