CODEWITHSUNDEEP
javascriptnode.jsasynchronouscallback
Jackson Darrow
Jackson Darrow

Reputation: 30

Calling a variable amount of asynchronous functions that are dependent on each other

I am trying to paginate through a website, where in order to request the next page, the information on the current page is required. 

function paginateSite(numPages, cb) {
    var stringToGetToNextPage = ''
    for (var i = 0; i < numPages; i++) {
        (function(i, stringToGetToNextPage) {
            request({
                url: 'http://website.com',
                qs: {
                    nextPage: stringToGetToNextPage
                }
            }, 
            function(err, res, body) {
                body_json = JSON.parse(body)
                console.log(body_json.stringToGetToNextPage)
            })
        })(i, stringToGetToNextPage)
    }
}

As you can see there is no way for me to set stringToGetToNextPage outside of the anonymous function. I have looked through the async library documentation but couldn't find anything that solved my problem.

Has anybody ran into this problem? It seems like this is a case where the async paradigm doesn't have a clean solution.

EDIT: QUESTION #2, is there conceivably a way to do this without recursion? Recursion is fine for small examples, but now I have a stackoverflow case that can kill the program without warning (and eats up memory).

Upvotes: 0

Views: 45

Answers (2)

JLRishe
JLRishe

Reputation: 101690

Recursion is the way to do this.

First, factor out a function to retrieve a single page:

function requestPage(stringToGetToNextPage, callback) {
    request({
        url: 'http://website.com',
        qs: {
            nextPage: stringToGetToNextPage
        }
    }, function(err, res, body) {
        if (err) { callback(err); }
        else { callback(null, JSON.parse(body)); }
    });
}

Then, make a function that can recursively call itself to get each successive page:

function requestAllPages(stringToGetToNextPage, callback) {
    requestPage(stringToGetToNextPage, function (err, page) {
        if (err) { callback(err); }
        else if (page && page.stringToGetToNextPage) {
            requestAllPages(page.stringToGetToNextPage, callback);
        }
        else { callback(); }
    });
}

Then kick it all off:

requestAllPages(null, function () {
    console.log('Done!');
});

Edit If you really do want to paginate a specific number of pages rather than going to the end, you can accomplish that with a small modification to requestAllPages:

function requestPages(numPages, stringToGetToNextPage, callback) {
    if (numPages <= 0) { callback(); return; }

    requestPage(stringToGetToNextPage, function (err, page) {
        if (err) { callback(err); }
        else if (page && page.stringToGetToNextPage) {
            requestPages(numPages - 1, page.stringToGetToNextPage, callback);
        }
        else { callback(); }
    });
}

requestPages(20, null, function () {
    console.log('Done!');
});

Upvotes: 1

Quentin
Quentin

Reputation: 943569

Write a recursive function.

function get_page(next_page_number, last_page_number, string_needed, callback) {

    request(
        {
            url: 'http://website.com',
            qs: {
                nextPage: string_needed
            }
        }, 
        function(err, res, body) {
            // NB: Bad variable name. `body` is JSON. `body_json` won't be.
            var body_json = JSON.parse(body);

            if (next_page_number == last_page_number) {
                callback(body_json);
            } else {
                get_page(
                    next_page_number + 1, 
                    last_page_number,
                    body_json.stringToGetToNextPage,
                    callback
                );
            }
        }
    );

}

get_page(1, 27, "", function (body_json) { console.log(body_json); });

Upvotes: 1

Related Questions