CODEWITHSUNDEEP
pythonmongodbmongoengine
Benjamin Andersen
Benjamin Andersen

Reputation: 482

How to do a Bulk Insert with MongoEngine?

I am struggling to do a Bulk Insert with MongoEngine. I can easily do a .save() with a loop to write to the database, however I cannot find the procedure to do a Bulk Insert from the MongoEngine documentation. The only mentioning was here on SO: multi document insert using mongoengine into mongodb. I tried to do it this way but I just insert all the documents without validating them in the Schema.

This is my code: 

import mongoengine as me

# connect to MongoDB
me.connect(host='CONNECTION_STRING')

# venue Schema
class newVenues(me.Document):
  name = me.StringField(required=True)
  latitude = me.DecimalField(required=True)
  longitude = me.DecimalField(required=True)


# some test data
venues = [{'name': 'Vega',
           'latitude': 55.672867,
           'longitude': 12.473692},
          {'name': 'Tivoli',
           'latitude': 55.681256,
           'longitude': 12.553412}]

# the list of venues to bulk insert to MongoDB
venues_to_insert_list = []
for venue in venues:
    venues_to_insert_list.append(
    newVenues(
    name=venue.get('name'),
    latitude=venue.get('latitude'),
    longitude=venue.get('longitude')
  )
)

# bulk insert to MongoDB
newVenues.objects.insert(venues_to_insert_list)

Upvotes: 2

Views: 2261

Answers (2)

Mustafa Khalid
Mustafa Khalid

Reputation: 191

import mongoengine as me

# connect to MongoDB
me.connect(host='CONNECTION_STRING')

# venue Schema
class newVenues(me.Document):
    name = me.StringField(required=True)
    latitude = me.DecimalField(required=True)
    longitude = me.DecimalField(required=True)


# venue Schema
class newVenues(Document):
    name = StringField(required=True)
    latitude = DecimalField(required=True)
    longitude = DecimalField(required=True)


# some test data
venues = [
    {"name": "Vega", "latitude": 55.672867, "longitude": 12.473692},
    {"name": "Tivoli", "latitude": 55.681256, "longitude": 12.553412},
]

# the list of venues to bulk insert to MongoDB


venues_to_insert_list = [newVenues(**data) for data in venues]

# bulk insert to MongoDB
newVenues.objects.insert(venues_to_insert_list)

Upvotes: 1

Shubham gupta
Shubham gupta

Reputation: 203

If you want to insert all the documents without validating them in the Schema. You need to create dynamic document in mongoengine

Below code is for read csv file and insert in mongodb in collection without validate schema [schema is also make in dynamic way].

import mongoengine as me
class DynamicDoc(me.DynamicDocument):
    any_field = me.StringField()

import pandas as pd

all_csv_records = data_frame.to_dict('records')
data_frame = pd.read_csv(file_path)
for data in all_csv_records:
    report_data = DynamicDoc()
    DynamicDoc.any_field = str('temp_data')
    for col, row in data.iteritems():
        report_data[col] = row
    report_data.save()

Note - Only 'any_field' in mongo model is necessary input manullay in string form. Because field define manually in mongo model.

save id mongodb like: enter image description here

Upvotes: 0

Related Questions