CODEWITHSUNDEEP
mongodbmeteor
Elie Steinbock
Elie Steinbock

Reputation: 5108

Long running Mongo queries in Meteor

How would one go about updating 1000s of documents in a collection in meteor where forEach has to be used to first calculate the changes for each individual document?

There's a timeout of 10 minutes or so as well as a certain number of megabytes. What I've done in the past is split the updates into groups of 300 and update like that. But is there a simpler way to do it in meteor to allow the for each loop to run for an hour of needed?

Upvotes: 1

Views: 132

Answers (1)

Michel Floyd
Michel Floyd

Reputation: 20256

Using percolate:synced-cron you could easily do this in batches.

SyncedCron.add({
  name: 'Update mass quantities',
  schedule: function(parser) {
    // parser is a later.parse object
    return parser.text('every 1 minute'); // or at any interval you wish
  },
  job: function() {
    var query = { notYetProcessed: true }; // or whatever your criteria are 
    var batchSize = { limit: 300 }; // for example
    myCollection.find(query,batchSize).forEach(doc){
      var update = { $set: { notYetProcessed: false }}; // along with everything else you want to update
      myCollection.update(doc._id,update); 
    }
  }
});

This will run every minute until there are no more records to be processed. It will continue running after that of course but won't find anything to update.

Upvotes: 2

Related Questions