CODEWITHSUNDEEP
javascriptnode.jspromise
Ollie
Ollie

Reputation: 1134

Running a promise.all of promise.alls

I have am making 400ish requests to a server - and putting each inside promises.

When running all 400 requests in a single promise.all - the system falls over.

I've split my requests into batches of 50 promises (and added them inside a promise.all), and added those all into another promise.all.

How can I run the promises in the batches, and wait for those to be done before moving onto the next?

// attach the other accounts a user has to the wrapper object
// **this is 400+ requests object, that has the requests in it**
// results are promises
const influencerAccounts = wrapper.map(p => addInfluencerAccounts(p));

// split the requests into chunks to stop the server falling over
const chunkedPromises = _.chunk(influencerAccounts, 50);

// promise.all on each chunk of promises/requests
// ????


// ...

return

I've tried looping over the chunked promised arrays (which is an array of promises) and Promise.all(ing) each one - but that's not going to wait from the previous batch to finish before sending the next.

Thanks,

Ollie

Upvotes: 2

Views: 2061

Answers (1)

T.J. Crowder
T.J. Crowder

Reputation: 1074108

You're making a mistake a lot of people make at first: Promise.all doesn't run anything. It just waits for things that are already running. By the time you've broken your influencerAccounts array into chunks, you've probably already overloaded the server, because you're still sending it 400+ requests at the same time.

Instead, chunk the payout array, and then process it in chunks, something along these lines:

const results = [];
const promise =
    _.chunk(payout, 50).reduce(
        (p, chunk) =>
            p.then(chunkResults => {
                results.push(...chunkResults);
                return Promise.all(chunk.map(startRequest)); 
            })
        ,
        Promise.resolve([])
    )
    .then(() => results);

I've used startRequest above instead of createInfluencerWrapper and addInfluencerAccounts because it wasn't clear to me if you'd introduced one or the other in an attempt to make your chunking work. But if not, startRequest is simply addInfluencerAccounts(createInfluencerWrapper(entry)).

That starts a chunk of 50 requests, uses Promise.all to wait for all of them to complete, then starts the next chunk of 50 requests. The "do this then when it's done do that" part comes from the promise reduce idiom, which in its simple form looks like this:

someArray.reduce((p, entry) => p.then(() => doSomethingWith(entry)), Promise.resolve());

It starts with a resolved promise, and hooks a then handler on it to do the next thing, which hooks a then handler on that to do the next thing, etc.

If you don't like closing over results, we can pass it along the reduce chain; here's the first version above doing that:

const promise =
    _.chunk(payout, 50).reduce(
        ({p, results}, chunk) => ({
            p: p.then(chunkResults => {
                results.push(...chunkResults);
                return Promise.all(chunk.map(startRequest)); 
            }),
            results
        }),
        {p: Promise.resolve([]), results: []}
    )
    .then(({results}) => results);

Upvotes: 2

Related Questions