CODEWITHSUNDEEP
mongodbmongodb-java
GJain
GJain

Reputation: 5093

mongodb bulk updates/inserts

We run mongo-2.4 via heroku/mongolab which does not support 2.6 for production

http://docs.mongolab.com/ops/#version-mgmt

One of my use, I wanted to do bulk inserts using java-driver:

// 1. Ordered bulk operation
BulkWriteOperation builder = coll.initializeOrderedBulkOperation();
builder.insert(new BasicDBObject("_id", 1));
builder.insert(new BasicDBObject("_id", 2));
builder.insert(new BasicDBObject("_id", 3));

BulkWriteResult result = builder.execute();

However, I can not upgrade prod to 2.6 and 2.4 does not support bulk inserts.

I could have upto many thousands inserts to mongo at any given time.

If I insert 1 document at a time, there could be performance/scaling issues.

Is there an alternate way of doing bulk inserts/upserts/updates in MONGODB in 2.4

Upvotes: 0

Views: 1711

Answers (2)

SUNDARRAJAN K
SUNDARRAJAN K

Reputation: 2295

you can bulk insert in mongo shell, keep all the insert documents into an array and then insert.

db.insert({"name": "Admin","project": "pro1"},{"name": "Admin","project": "pro2"});

Upvotes: 1

Rob Moore
Rob Moore

Reputation: 3383

Short answers:

  • No for upserts/updates.
  • Yes for inserts and it will happen automatically if you use the BulkWrite API.

Some details:

For a pre-2.6 MongoDB there is no way to send a batch of updates to the server. You have to use the OP_UPDATE message which only supports a single update operation at a time.

The good news is that the Java driver will degrade to using the old messages automatically based on the server version it is connected to. For inserts this will still result in batches since OP_INSERT supported batches pre-2.6.

This means you can write your code to use batching and for now it will perform close to optimal (but still incur the per-request round trip delays). When MongoLab supports 2.6 you won't have to change your code and the update batching will just happen.

You mention batching "many thousands" of operations. Realize that there are limits on the size of the batches. Currently they are 1000 operations and a little over 16MB. I have advised people to limit batches to a couple of thousand as anything larger has no impact on performance and even at 2000 must be broken up by the driver before being submitted to the server.

Lastly, if you really need to get rid of the round-trip latency you can look at using the Asynchronous Java Driver. If you are willing to do a little work to allow more requests to be in-flight at once you can achieve most if not all of the performance from batching.

A fairly easy strategy is to use a pending results queue to hold the futures for the in-flight requests. You start by sending requests as fast as possible and store the Future for each request in a blocking queue. When the queue is full you then alternate between sending requests and processing the results. Once all of the requests have been sent you simply drain the results queue.

This simple strategy can greatly improve the throughput of an application.

HTH, Rob.

P.S. In full disclosure I work on the Asynchronous Java Driver.

Upvotes: 3

Related Questions