CODEWITHSUNDEEP
mongodbmongodb-query
dan dolev
dan dolev

Reputation: 31

mongoDB bulkWrite bad performence

I have MongoDB collection that contains around 10M documents

I try to update (upsert) around 2500~ documents, each update is around 1K

I try to use bulkWrite with order=false. It took around 10 seconds and it like 3-4 ms per document.

So I try to insert the 2500~ documents by updateOne (iterative mode) I measure the avg time per document and it took like 3.5ms for each update.

Why I don't get a better result for bulkWrite and how can I improve the bulkWrite update time?

Example for bulkUpdate with 1 document:

    db.collections.bulkWrite( [
   { updateOne :
      {
         "filter": {"Name": "someName", "Namespace" : 
         "someNs", "Node" : "someNode" , "Date" : 0},
         "update": {"$addToSet" : {"Data" :{"$each" : ["1", , "2"]}}},           
         "upsert": true
      }
   }
] )

Document Example:

{
  "Name": "someName",
  "Namespace": "someNs",
  "Node": "SomeNode",
  "Date": 23245,
  "Data" : ["a", "b"]
}

I have a compound index that contains: Name, Namespace, Node, Date.

When I try to find documents I get good results

Upvotes: 0

Views: 2194

Answers (1)

Yahya
Yahya

Reputation: 3444

TL DR; play around with your batch sizes to find the sweet spot.

Bulk write or updateMany will be faster than single updates. Simply because there is less chatter going on (less round trips). The amount of data being transferred is exactly the same.

What you need to do is to find an optimum number which gives you the highest throughput based on your network, cluster config etc.

Typically what you'd see is that if batch size is smaller, you are not using it to the ability. And if it is too big, then you are spending too much time just transferring the package to database.

Upvotes: 3

Related Questions