CODEWITHSUNDEEP
javascriptnode.jsasynchronous
ton1
ton1

Reputation: 7628

Function iteration with Async waterfall in Node.js

I am trying to get resource from the some url, that resource is separated as multiple pages, the number of page is over 500, but the order of resource have to be guaranteed, so I decided to use Async module. 

function getData(page, callback){
    var url = "http://someurl.com/document?page="+page;
    // get data from the url ...
    // when success, 
    callback();
};

So, above function is get resource from some url, I have to iterate this function to many times, but I don't know how can I iterate this with async waterfall. What point I should push the for iteration? 

async.waterfall([

    // page 1 
    function(callback){
        getData(1, function(){
            callback(null);
        });
    },

    // page 2
    function(callback){
        getData(2, function(){
            callback(null);
        });
    },

    // page 3, 4, 5..... 100, ... 500


],function(err, result){
    if(err) return next();

    console.log('finish !');
});

Upvotes: 0

Views: 510

Answers (3)

Nicholas Robinson
Nicholas Robinson

Reputation: 1419

Why don't you use Promises:

function getData(page, callback){
    var url = "http://someurl.com/document?page="+page;
    // var request = call to get data from the url ... 
    return request;
};

var promises = [];

for (var page = 1; page < totalPages; page++) {
    promises.push(getData(page));
}

Promise.all(promises).then(function (listOfResults) {
    // Do callback here
});

Just make sure that your method of sending the get request returns a promise. If not you can use a function like this:

function usesCallback(data, callback) {
    // Do something with data
    callback();
}

function makePromise(data) {
    return new Promise(function (resolve, reject) {
        usesCallback(data, resolve);
    });
}

Here is some more info on Promises

Upvotes: 2

lujjjh
lujjjh

Reputation: 473

If you do want to use async, async.map() and async.mapLimit() are better choices since they apply the iteratee to each item in parallel and guarantee the results array to be in the same order as the original collection.

async.mapLimit(_.range(1, 500), 10, getData, function (err, results) {
    // do something with results
});

The code above means to get data from page 1 to page 500, with no more than 10 async operations at a time.

_.range(1, 500) is a function from underscore to generate an array [1, 2, 3, ..., 500]. If you do not like to use underscore in your project, you can simply rewrite it such as:

function range(lower, upper) {
    return Array.apply(null, Array(upper - lower + 1))
        .map(function (_, i) { return lower + i; });
}

Upvotes: 1

Harekam Singh
Harekam Singh

Reputation: 66

You can use async's whilst for this check here.

Upvotes: 0

Related Questions