CODEWITHSUNDEEP
javascriptnode.jsmongodbasynchronous
Bharat
Bharat

Reputation: 2479

MongoDB and Node js Asynchronous programming

I am trying to solve an exam problem, so I cannot post my exam code as it is. So I have simplified such that it addresses the core concept that I do not understand. Basically, I do not know how to slow down node's asynchronous execution so that my mongo code can catch up with it. Here is the code:

MongoClient.connect('mongodb://localhost:27017/somedb', function(err, db) {
  if (err) throw err;
  var orphans  = [];
  for (var i; i < 100000; i++) {
    var query = { 'images' : i };
    db.collection('albums').findOne(query, function(err, doc_album) {
        if(err) throw err;
        if (doc_album === null) {
          orphans.push(i);
        }
    });
  }
  console.dir(orphans.length);
  return db.close();
});

So I am trying to create an array of those images who do not match my query criteria. I end up with a orphans.length value of 0 since Node does not wait for the callbacks to finish. How can I modify the code such that the callbacks finish executing before I count the number of images in the array that did not meet my query criteria?

Thanks in advance for your time.

Bharat

Upvotes: 2

Views: 5625

Answers (4)

Andrey Sidorov
Andrey Sidorov

Reputation: 25466

I assume you want to do 100000 parallel DB calls. To "wait" 10000 calls completion in each call callback we increase finished calls counter and invoke main callback when last one finished. Note that very common mistake here is to use for loop variable as a closure inside callback. This does not work as expected as all 10000 handlers scheduled first and by the time first is executed loop variable is of the same, maximum value.

function getOrphans(cb) {
  MongoClient.connect('mongodb://localhost:27017/somedb', function(err, db) {
    if (err) cb(err);
    var orphans  = [];

    var numResponses = 0;
    var maxIndex = 100000
    for (var i = 0; i < maxIndex; i++) {
       // problem: by the time you get reply "i" would be 100000.
       // closure variable changed to function argument:  
       (function(index) {
        var query = { 'images' : index };
        db.collection('albums').findOne(query, function(err, doc_album) {
          numResponses++;
          if(err) cb(err);
          if (doc_album === null) {
            orphans.push(index);
          }
          if (numResponses == maxIndex) {
            db.close();
            cb(null, orphans);
          }
        });
      })(i); // this is "immediately executed function
    }
  });
 }


 getOrphans(function(err, o) {
   if (err)
     return console.log('error:', err);
   console.log(o.length);
 });

Upvotes: 2

6502
6502

Reputation: 114579

Im not suggesting this is the best way to handle this specific problem in Mongo, but if you need to wait to the DB to reply before continuing then just use the callback to start next request.

This is not obvious at first, but you can refer to the result processing function inside the function itself:

var i = 0;
var mycback = function(err, doc_album) {
    // ... process i-th result ...
    if (++i < 100000) {
        db.collections("album").findOne({'images': i}, mycback);
    } else {
        // request is complete, "return" result
        result_cback(null, res);
    }
};
db.collections('album').findOne({'images': 0}, mycback);

This also means that your function itself will be async (i.e. will want a result_cback parameter to call with the result instead of using return).

Writing a sync function that calls an async one is just not possible.

You cannot "wait" for an event in Javascript... you must set up an handler for the result and then terminate.

Waiting for an event is done in event-based processing by writing a "nested event loop" and this is for example how message boxes are handled in most GUI frameworks. This is a capability that Javascript designers didn't want to give to programmers (not really sure why, though).

Upvotes: 1

Josh C.
Josh C.

Reputation: 4373

You don't need to slow anything down. If you are simply trying to load 100,000 images from the albums collection, you could consider using the async framework. This will let you assign tasks until the job is complete.

Also, you probably don't want request 100,000 records one-by-one. Instead, you probably want to page them.

Upvotes: 0

painotpi
painotpi

Reputation: 6996

Since you know it does not wait for the call to come back. You can do the console.dir inside your callback function, this should work (although I haven't tested it) 

    db.collection('albums').findOne(query, function(err, doc_album) {
        if(err) throw err;
        if (doc_album === null) {
          orphans.push(i);
        }
        console.dir(orphans.length);
    });

Upvotes: 0

Related Questions