CODEWITHSUNDEEP
node.jsmongodb
Deepika P.
Deepika P.

Reputation: 1959

Run MongoDB query in loop synchronous in nodejs

I'm newbie to NodeJS and MongoDB and I'm stuck in a problem where I have to run a query in loop and the response of query will decide if loop has to be continued.

I have to call an API and pass offset as parameter in it. The API returns 50 records at a time. So, when I get 50 records I increment offset to 51 and call the same API again with offset 51 to get next 50 records. This will go on until API gives less than 50 records. This is the point when I can exit. So, I don't have anything well in advance that can define how long the loop will iterate. It depends on previous API response. In each API call I save the data fetched in my db(MongoDB).

Here is my rough code-

var Client = require('node-rest-client').Client;
var client = new Client();
var MongoClient = require('mongodb').MongoClient;
var dburl = "mongodb://localhost:27017/mydb";
var repeat  =   true;
var result;
while(repeat){
    client.get('http://example.com?$offset='+offset, function (data, response) {
         result         =   JSON.parse(data);           
         num_records    =   ret.length;
         MongoClient.connect(dburl, function(err, db) {
             if (err) throw err;
             var collection = db.collection('myCollection');      
             collection.insertMany(result, function(err, res) {
                if (err) throw err;
                if(num_records < 50){
                     repeat =   false;
                }
                db.close();
             });      
          });
      });
      if(offset==0)
        offset  = 51;
      else
        offset  += 50;
}

This loops infinitely. Please help!

Upvotes: 0

Views: 1586

Answers (1)

boehm_s
boehm_s

Reputation: 5534

Can you tell me in the comment why do you need to get your records 50 by 50 ?

First thing : you're mixing synchronous and asynchronous code : 

var i = 0;
while (true) {
  client.get(url + offset, function(data, response) {
    /* deal with your response */
    console.log("hey, I'm asynchronous !");
  }
  console.log("i : " + i++);
}

Here you will have this kind of output : 

// i : 1
// i : 2
// i : 3
// hey, I'm asynchronous !
// i : 4

The code that is in your callback function will not stop your loop. If you need to understand why, - and how it works in JS, read this.

So, you can't put async code in your loops, it will alaways end badly.
But another way to loop is recursion, so here's what you can do : 

function get50records(url, offset, callback) {
  DoYourrequests(_exemple, _exemple, function(requestResult) {
    if (youNeedToContinue) {
      console.log("offset : " + offset);
      get50records(url, offset + 50, callback);
    } else
      callback("end Message !");
  })
}

get50records(yourUrl, 50, function(res) {
  console.log(res);
  // continue to code here
})

So, each time you get the result of your request, if you need to continue, the function will call itself with the new offset and if you don't need to continue, it will call the callback function that you passed in 3rd parameter.

Here is what the logs would look like : 

// offset : 50
// offset : 100
// offset : 150
// offset : 200
// end Message !

Once you master the callback pattern, you can learn how the Promise object works, and then use thins like async / await

Upvotes: 1

Related Questions