roger
roger

Reputation: 9893

mongoengine bulk upsert a batch of record?

I want to upsert(update or insert) a list of record, in fact I know that mongodb support bulk operation since mongodb3.0.

I want to know whether mongoengine support bulk upsert operation in mongoengine (0.10.0).

if not, I want to know how to upsert a list of record, I know mongoengine support insert batch like this:

class User(Document):
    username = StringField(required=True)
    password = StringFiedl(required=True)
    meta = {'db_alias': 'user_info', 'collection': 'user', 
             'indexes': [{'fields': ['username'], 'unique': True}]
           }

def save_users(self, users):
    Users.objects.insert(users) # raise mongoengine.errors.NotUniqueError

Upvotes: 3

Views: 3561

Answers (2)

warvariuc
warvariuc

Reputation: 59604

You can use mongo.collection.Collection.bulk_write:

operations = [
    pymongo.ReplaceOne({'username': user. username}, user.to_mongo(), upsert=True)
    for user in users
]
result = User._get_collection().bulk_write(operations)

Upvotes: 2

Blakes Seven
Blakes Seven

Reputation: 50406

You can actually use the bulk operations API directly by accessing the underlying collection object from the pymongo driver MongoEngine uses. MongoDB itself has supported bulk operations since version 2.6. There are newer methods to access these since v3 of the pymongo driver, but the basic methods have been around since the corresponding driver update to the 2.6 Server version ( pymongo 2.7 ).

To get this with MongoEngine you can call the undocumented ._get_collection() from your class to return the collection object:

bulk = Users._get_collection().initialize_ordered_bulk_op()

for user in users:  # where users is a list of dicts containing data to work on
    bulk.find({ "matchField": user['matchField'] }).upsert().replace_one(user)

bulk.execute()

Or any other usage of bulk methods such as .update_one() which you probably want. And .upsert() is a chained method that modifies such update statements.

You are working with raw python objects here as there is no direct equivalent in MongoEngine itself. But you can use the operations by accessing the methods from the underlying driver

Upvotes: 7

Related Questions