CODEWITHSUNDEEP
node.jsasynchronous
Ugur
Ugur

Reputation: 2024

Iterate over async function

I have a function that uses the module cherio to get data from a website.

Now I'd like to iterate this function over an array of keywords, collect the intermediate results in an array named stats and finally print the results of the stats array to the console via console.log()

Whenever I run this script it triggers the async function quickly and prints an empty stats array.

Now my question: How can I wait for the async functions to complete so that I can print the array to console when it's populated / finished.

I have googled a lot and searched stack overflow. There seem to be many ways to accomplish my goal, but what is the most idiomatic way in node to do this?

Here is the way I solved it:

var request = require("request"),
    cheerio = require("cheerio"),
    base_url = "http://de.indeed.com/Jobs?q=";  // after equal sign for instance:   sinatra&l=

/* search syntax:
   - http://de.indeed.com/Jobs?q=node&l=berlin&radius=100
   - 
   - 
*/ 

// //
var search_words = ["django", "python", "flask", 
                    "rails", "ruby",
            "node", "javascript", "angularjs", "react",
            "java", "grails", "groovy",
            "php", "symfony", "laravel"
            ];

var counter = 0;
var stats = [];


function getStats(keyword) {
    url = base_url + keyword + "&l=";
    request(url, function(err, resp, body) {
    if(!err) {
        $ = cheerio.load(body);
        data = $("#searchCount")[0].children[0].data.split(" ").reverse()[0];

        stats.push([keyword, data]);
        counter++;
    }
    // list complete?
    if (counter === search_words.length) {
        console.log(stats);
    }
    });
}

for (var j=0; j<= search_words.length; j++) {
    getStats(search_words[j]);
}

Upvotes: 2

Views: 1200

Answers (3)

Lucas Manzke
Lucas Manzke

Reputation: 128

The most common way I can think of is using a promise library like Q.

npm install --save q

Then use it in your code:

var Q = require('q');
var requestFn = q.denodeify(request);

Then you iterate over your values:

var promises = search_words.map(function(keyword) {
   url = base_url + keyword + "&l=";
   return requestFn(url);
});

Q.all(promises).then(function(values) {
    //values will contain the returned values from all requests (in array form)
}, function(rejects) {
   //rejected promises (with errors, for example) land here
});

The denodeify function from Q basically turns the callback-based function into one that returns a promise (a step-in for the future value, as soon as it's there). That function is requestFn (find a better name for it!). All these promises are collected in one Array which is passed to Q.all to make sure that all promises are fulfilled (if one is rejected, other promises are rejected too).

If that is not your intended behavior: There are loads of ways to play with the excellent Q library. See the documentation: https://github.com/kriskowal/q

I did not bullet proof test this code. You might need to play around with it a bit, but it should give you a good idea of how to soundly do things like this. Having a counter go up is usually a very unreliable way of handling asynchronous code.

Upvotes: 1

Lewis
Lewis

Reputation: 14866

Promise is the best solution for handling asynchronous operations.

Promise.all(search_words.map(function(keyword) {
  return new Promise(function(resolve, reject) {
    request(base_url + keyword + "&l=", function(err, resp, body) {
      if (err) {
        return reject(err);
      }
      $ = cheerio.load(body);
      resolve([keyword, $("#searchCount")[0].children[0].data.split(" ").reverse()[0]]);
    });
  });
})).then(function(stats) {
  console.log(stats);
});

Upvotes: 3

user1092803
user1092803

Reputation: 3277

Other the usual (and correct) way you use to solve the problem, there are some modules that let you write code that is synchronous, if you really want.

Try to google for "nodejs synchronous" give as result some link to nodejs modules and/or methodologies to write synchronous code in nodejs, but I suppose they are usefull only to some specific problem (never used them myself)

Upvotes: 0

Related Questions