CODEWITHSUNDEEP
pythonmongodbpython-2.7pymongomongoengine
Kumar Nitin
Kumar Nitin

Reputation: 1863

How to create 100000 entries in mongodb at a time?

While creating 100000 (that's One Hundred Thousand) entires at a time, it crashes. My DB is something like this:

EmbDoc(EmbeddedDocument):
    id = StringField(required=True, primary_key=True)
    field_one = StringField()
    ...

MyDoc(Document):
    emb_doc_list = EmbeddedDcoumentListField(EmbDoc)
    total_emb_doc = IntField()
    ...

Now I want each of MyDoc to have upto 100 EmbDoc and if it reaches 100, I create another MyDoc.(My actual data is present in EmbDoc). In order to do this, I query MyDocs and find the existing MyDoc which has less than 100 entires and add an entry.

This logic works just fine if I try to create 1000 entires(but takes around 40 second on local machine). But when I try to add large data at a time, it fails miserably.

Any idea on how to solve this problem will be helpful.

Upvotes: 3

Views: 319

Answers (1)

Kumar Nitin
Kumar Nitin

Reputation: 1863

The problem was with the db query and serialization. Since we were querying all the MyDoc every time, the performance was affected.
We started maintaining the hash map for MyDoc id and total_emb_doc. Now we just do one look up and get the Doc.
Currently it is taking 10 seconds for every 10000 entries(with serialization).

Upvotes: 1

Related Questions