CODEWITHSUNDEEP
node.jsmongodb
Vikash Kumar
Vikash Kumar

Reputation: 19

How to insert 5 million records in MongoDB?

I am here trying to insert 5 million documents in MongoDB and it is taking too much of time. Please suggest me a more efficient way. I have here posted db structure and related query (code). Please have a look at it and point out improvements that can be done to speed up the insertion.

DB structure:

    { 
        "_id" : {
            "msisdn" : "919899587091", 
            "op" : "idea", 
            "eid" : "547c0a0ccbbc64ce2b773488", 
            "cid" : "547c0a8ecbbc64cf2b773488", 
            "d" : ISODate("2015-05-26T04:30:00.000+0000")
        }, 
        "value" : {
            "unq" : NumberInt(1), 
            "ut" : "NNN"
        }
    }

Up to 5 million.

I am using aggregate as given below: 

    var v = db.collection(collectionname).aggregate([{$match:{'@timestamp':             {'$gte':new Date(starttime), '$lt':new Date(endtime)},'eid':{$ne:'-'},ut:{$ne:'-'}}},
        {$project:{'msisdn':"$msisdn",'op':'$op','eid':'$eid','cid':'$cid','ut':'$ut','mi':{'$millisecond':'$@tim`enter code here`estamp'},'m':{'$minute':'$@timestamp'},'s':{'$second':'$@timestamp'},d:'$@timestamp'}},
        {$project:{'msisdn':"$msisdn",'op':'$op','eid':'$eid','cid':'$cid','ut':'$ut','d':{'$subtract':['$d',{'$add':[{'$multiply':['$m',60,1000]},{'$multiply':['$s',1000]},'$mi',1800000]}]}}},
        {$group : {'_id' : {'msisdn':"$msisdn",'op':'$op','eid':'$eid','cid':'$cid','d':'$d'},'unq':{$sum:1},'uts':{$addToSet:'$ut'}}},
        {$project:{'_id':'$_id','value':{'unq':'$unq',
      'ut':{ $cond: {if: {$setIsSubset:[['NNN'],'$uts']} ,then: 'NNN',
         else: { $cond: {if: {$setIsSubset:[['RNN'],'$uts']} ,then: 'RNN',
           else:{ $cond: {if: {$setIsSubset:[['RRN'],'$uts']} ,then: 'RRN',else:'RRR'}}
                }}
          }}} }
        }],{ allowDiskUse: true, cursor: {batchSize: 1000}});

    var bulk = db.collection(outcollectionname).initializeUnorderedBulkOp();

    v.on('data', function(data){ 
            bulk.insert(data);  
        });

    v.on('end', function(){ 
    bulk.execute(function(err, result) {

            });
        });

Upvotes: 3

Views: 1486

Answers (1)

B3rn475
B3rn475

Reputation: 1057

I this you are doing manually what the $out stage of the aggregation pipeline is for.

http://docs.mongodb.org/manual/reference/operator/aggregation/out/

It stores the results into another collection.

This avoids to copy the object into the nodejs context and copy it back into mongodb.

Upvotes: 3

Related Questions