CODEWITHSUNDEEP
mongoimport
user3407386
user3407386

Reputation: 172

mongoimport very very slow for large json file

I have a large json file (350GB) and I am trying to import it in MongoDB collection using mongoimport.The mongoimport is very very slow and I am not sure how many days it will take.

Can any one please suggest the best way to load this json file to mongodb collection. I have enough disk space to load this json file.

Upvotes: 3

Views: 4973

Answers (3)

sandhya murugesan
sandhya murugesan

Reputation: 114

Use a Studio3T-mongoChef GUI client where importing JSON,dump etc. are simple yet faster.

Upvotes: 1

doctori
doctori

Reputation: 352

If you are using mongodb > 3.0.0 you can use the --numInsertionWorkers on the mongoimport command
Set this to the number of CPUs you have in order to speedup the import.
ref.

Upvotes: 3

user2829759
user2829759

Reputation: 3512

I came across similar situation. I used mongorestore instead of mongoimport but the idea is the same. iotop shows that the restore process had an IO rate of about 1M/s which is pretty low. As other post here suggests, the low performance is probably due to the json to bson serialization. So I ended splitting up the exported json file into different chunks with the following command

mongodump --host < host > --port < port > --username < user > --password < pwd > --authenticateionDatabase admin --db < db > --collection < coll > --query "{DayOfWeek:"Monday"}" --out "SomeDir-Monday" &

mongodump --host < host > --port < port > --username < user > --password < pwd > --authenticateionDatabase admin --db < db > --collection < coll > --query "{DayOfWeek:"Tuesday"}" --out "SomeDir-Tuesday" &

...

then I ended up having 7 chunks. Finally import these chunks in parallel using mongorestore with following command.

mongorestore --host < host > --port < port > --username < user > --password < pwd > --authenticateionDatabase admin --db < db > --collection < coll > PATH_TO_MONDAY.json &

mongorestore --host < host > --port < port > --username < user > --password < pwd > --authenticateionDatabase admin --db < db > --collection < coll > PATH_TO_TUESDAY.json &
...

Upvotes: 3

Related Questions