CODEWITHSUNDEEP
javascriptnode.jsrecursionpromiseout-of-memory
J Deane
J Deane

Reputation: 47

Iterating through expensive async function - memory constraints, recursion?

I'm having some issues with what I think is a basic problem regarding use of Promises in my node.JS server side application - unfortunately I can't see how to resolve it despite seeing other similar questions (I think).

Basically my issue is this: I am trying to retrieve some external data and then process it. There is a lot of data so I have to retrieve it page by page. Additionally given the size of the data, my server cannot execute multiple calls/processes at once as I run out of memory and the server crashes. I don't know until execution time how many pages I have to retrieve to get all the data.

I have tried executing a forEach loop with an array of the number of pages however this clearly doesn't work. e.g.:

pages = [1,2,3,4];
pages.forEach( function(pageNumber){
  veryMemoryExpensiveFunctionRetrievingAndProcessingPage(pageNumber).then(
// handle the results);
})

(the behaviour here is that all functions execute synchronously and the server runs out of memory).

I'm pretty stuck here - I know I need to execute that function multiple times synchronously but dont know where to start with doing so! I've also attempted recursion however this again causes out of memory as each call adds to the stack.

Upvotes: 1

Views: 67

Answers (1)

user1726343
user1726343

Reputation:

You need to obtain a promise for each page when the previous one completes, rather than all at once. i.e

function fetchAndProcessPages(i, handlePage) {
    retrievePage(pages[i]).then(page => {
        handlePage(page);
        if (i+1 < pages.length) fetchAndProcessPages(i+1, handlePage);
    });
}
fetchAndProcessPages(0, page => console.log(page));

Upvotes: 1

Related Questions