Copying a mongo collection with huge data to another collection?

Question

I have a mongo collection(s) with 2.5 million data and that may grow upto 3 million. I am using spring batch and am trying to copy that collection to another collection. Approaches I have used are as follows :

• Inside a tasklet, I have Created a ProcessBuilder object and called a shell script which executes a mongo query. Content of shell script is as follows :

  >  mongo $serverURL/$dbName js-file-to-execute.js
  
  // js file contains copy command (db.collection.copyto('newCollection'))
  

For less data (< 200 k) it works fine but for 2 million data it hangs the mongo server and the job got failed with Socket Exception

• Used a mongo template and executed a query

dbMongoTemplate.getDb().getCollection("collection").aggregate(Arrays.asList((DBObject) new BasicDBObject("$out","newCollection")));

This executes a mongo aggregate query db.collection.aggregate({$out : "newCollection"})

This also worked for collections with less data but for larger data set it keeps running until socket time out occurs and fails the job at the end.

Please suggest efficient way to copy data?

Answer 1

//Fastest way to copy a Collection in MongoDB
db.getCollection('OriginalCollection').aggregate([ { $out: "ClonedCollection" } ]);

This command copied a collection of 2 million records in about 2-3 minutes.

https://gist.github.com/tejzpr/ff37324a8c26d13fef08c318278c0718

Answer 2

To copy this collection I will sugest using mongodump/mongoexport

mongodump --db databaseName --collection collectionName --out directory-path

then copy directory directory-path and then restore on target machine using

mongorestore --db databaseName --collection collectionName directory-path