CODEWITHSUNDEEP
mongodbbulkinsert
VaidAbhishek
VaidAbhishek

Reputation: 6144

MongoDB Bulk Silent Update

I need to insert a set of documents in a remote MongoDB instance, in which some of these documents may already be residing at the instance i.e. having _id parameter. I want MongoDB to ignore such documents and commit the rest.

Currently the default behavior of PyMongo's insert method is that, it returns back a DuplicateError Exception for each of the duplicate document encountered at the remote server. I want to suppress this behaviour into a more fire and forget kind of functionality.

If anyone knows how to do this. It'll be appreciable.

Upvotes: 0

Views: 666

Answers (1)

Ross
Ross

Reputation: 18111

For bulk inserts where there may be duplicate key errors you need to set the continue_on_error flag.

With this flag inserts will continue even if an error has happened. The last error will be reported back by getLastError - which you can catch or if you want fire and forget set the write concern to 0.

from pymongo import *
client = MongoClient()
coll = client.test.test
coll.drop()

# Add a test document
coll.save({'_id': 1, 'hello': 'world'})
print(coll.count())

# Without the flag - Boom
coll.insert([{"_id": 1, "hello": "world"}, 
             {"_id": 2, "Howdy": "Worldy"}])
print(coll.count())

# With a write concern of 0 - no error but not saved.
coll.insert([{"_id": 1, "hello": "world"}, 
             {"_id": 2, "Howdy": "Worldy"}], w=0)
print(coll.count())

# Will error but will insert as well
coll.insert([{"_id": 1, "hello": "world"}, 
             {"_id": 2, "Howdy": "Worldy"}], continue_on_error=True)
print(coll.count())

# With a continue_on_error and write concern of 0 - will only error if theres 
# some socket or network error
coll.insert([{"_id": 1, "hello": "world"}, 
             {"_id": 2, "Howdy": "Worldy"},
             {"_id": 3, "Hi": "World"}], w=0,  continue_on_error=True)
print(coll.count())

Upvotes: 1

Related Questions