CODEWITHSUNDEEP
pythonmongodbpymongodatabase
RoboCop87
RoboCop87

Reputation: 837

MongoDB insert raises duplicate key error

I am getting the following error when trying to do a bulk insert into an empty mongodb collection.

pymongo.errors.DuplicateKeyError: E11000 duplicate key error index: cmdDistros.locDistro.$id dup key: { : ObjectId('51dac9d0c74cd81acd85c0fd') }

I am not specifying an _id when I create any of the documents, so mongodb should create the unique index correct? Here is the code I used:

#Populate database with uniform distribution
            entries = []
            for coor in freeIndices:
                for theta in range(360):
                    entry = {"x" : coor[0], "y" : coor[1], "heading" : theta}
                    for i in range(numData):
                            entry["data" + str(i)] = 1./numData
                    entries.append(entry)
            print "Entries created, loading into database..."

            locDistro.insert(entries)

Taking fate out of mongoDB's hands, I tried creating my own index using:

#Populate database with uniform distribution
            entries = []
            idNum = 0
            for coor in freeIndices:
                for theta in range(360):
                    print idNum
                    entry = {"_id" : idNum, "x" : coor[0], "y" : coor[1], "heading" : theta}
                    idNum += 1
                    for i in range(numData):
                            entry["data" + str(i)] = 1./numData
                    entries.append(entry)
            print "Entries created, loading into database..."

            locDistro.insert(entries, manipulate = False)

The print statement showed each idnum as the documents were created, and they were all unique and incremented just as expected. However on insert, I received the error:

pymongo.errors.DuplicateKeyError: E11000 duplicate key error index: cmdDistros.locDistro.$id dup key: { : 0 }

and only one document was inserted into my database.

I am completely stumped, anyone have an answer as to why this might be happening?

Upvotes: 19

Views: 37019

Answers (5)

Arslan Arif
Arslan Arif

Reputation: 335

SOLUTION: Declare dict() item inside the loop and then populate and insert it. I had a similar problem while using insert_one() from pymongo. I solved my problem by declaring the dict() item inside the loop. Here is the working version of your code:

#Populate database with uniform distribution
            entries = []
            for coor in freeIndices:
                for theta in range(360):
                    entry = dict()
                    entry['x'] = coor[0]
                    entry['y'] = coor[1]
                    entry['heading'] = theta
             
                    for i in range(numData):
                            entry['data' + str(i)] = 1./numData
                    entries.append(entry)
            print "Entries created, loading into database..."

            locDistro.insert(entries)

Upvotes: 1

Alfredo cubitos
Alfredo cubitos

Reputation: 171

I had the same error using insert_one() and also insert_many()

My solution is, to use update_one() with upsert=True

  doc = {a: 1, b:2, x:{xx:"hello",yy:"world"}}
  db.collection.update_one(doc,{'$set':doc},upsert=True)

This works for me :-)

Upvotes: 6

Safvan CK
Safvan CK

Reputation: 1340

Make sure variable 'entries' is cleared after every insert.

The problem is that PyMongo injects an _id field into the document, if the _id field does not exist, before inserting it (_id is always generated client side). That means that the first time through the loop _id is added by the insert method. Since 'entries' is defined outside, each subsequent pass through the loop uses the same value for _id.

Clear the dict variable in top of the loop statements.

OR

Remove _id from the dict. Eg:

del my_dict['_id']

Upvotes: 4

gaurhari dass
gaurhari dass

Reputation: 167

Delete the key "_id":

for i in xrange(2): 
    doc['i'] = i 
    if '_id' in doc: 
        del doc['_id'] 
    collection.insert(doc)

Or manually create a new one:

from bson.objectid import ObjectId

for i in xrange(2): 
    doc['i'] = i 
    doc['_id'] = ObjectId() 
    collection.insert(doc)

Getting "err" : "E11000 duplicate key error when inserting into mongo using the Java driver

Upvotes: 10

A. Jesse Jiryu Davis
A. Jesse Jiryu Davis

Reputation: 24017

You need to understand that your entries list has a bunch of references to one entry dict. So when PyMongo sets entries[0]['_id'], all the other entries get the same _id. (In fact, PyMongo will iterate through the list setting each entry's _id, so all the entries will have the final _id at the end.) A quick fix would be:

entries.append(entry.copy())

This is merely a shallow copy, but in the code you shared I believe this is enough to fix your problem.

Upvotes: 30

Related Questions