CODEWITHSUNDEEP
pythonmongodbcsvmongoimport
Manish Gupta
Manish Gupta

Reputation: 4656

python mongoimport error inserting documents lost connection to server

I am trying to insert a csv file to mongodb using python os module but i am getting this error:

2016-03-02T05:25:23.718+0000 connected to: localhost

2016-03-02T05:25:23.731+0000 error inserting documents: lost connection to server

2016-03-02T05:25:23.731+0000 Failed: lost connection to server

2016-03-02T05:25:23.731+0000 imported 0 documents

My code is :

def uploadBulkProducts(self, new_products_file):
    with open('new_products.csv','wb') as new_file:
        for chunk in new_products_file.chunks():
            new_file.write(chunk)

    try:
        os.system("mongoimport --db picknbox --collection temp_products --type csv --file new_products.csv --headerline")
        os.remove('new_products.csv')
        return True
    except:
        return False

I read this, this and this post. They all say that I should limit the batch size to lesser value than default (10000). But i have only 1 row in my csv file.

My CSV FILE

I tried the same from terminal but same. I am using MongoDB version 3.0.9

EDIT:

I am using django to import a csv file to a mongodb collection. I saved the file to a directory. and then import the file using mongoimport command.

I can save the file easily and i can see it in my directory. I confirmed that mongod is running . The error log is from django development server.

This code is working fine in another project running on same versions.

Also when i run the same mongoimport command from terminal i got the exact same error. I ran a read statement side by side in mongo shell and it worked fine.

Upvotes: 0

Views: 665

Answers (1)

Blakes Seven
Blakes Seven

Reputation: 50406

Your CSV is in UTF-16 format. On unix, convert the content to allow it as input:

iconv -f UTF-16 -t ISO-8859-1 new_products.csv \
   | mongoimport --db picknbox --collection temp_products --type csv --headerline

So the iconv utility will do that for you and you can just pipe the output directly into mongoimport, or convert otherwise to a format accepted by mongoimport.

Upvotes: 1

Related Questions