CODEWITHSUNDEEP
javascriptnode.jspromiserequest
JaapH
JaapH

Reputation: 391

Recommended way to issue multiple parallel REST calls and wait for all results in node.js

I want to call several REST APIs to obtain data I use. I want all the REST calls to finish and then do some calculation. The code below works but I do not like it that much. Especially when one site returns an error it is not that easy to see where it come from. I would prefer to handle each call separately and then in the end "wait" for all calls to complete. That way I could also do a partial calculation if somehow one REST call did not work 

const rp = require('request-promise');


var site1 = {
    uri: 'https://www.site1.com/api/v1/somedata.json',
    json: true
};


var site2 = {
    uri: 'https://www.site2.com/api/v1/somedata.json',
    json: true
};

var site3 = {
    uri: 'https://www.site3.com/api/v1/somedata.json',
    json: true
};


var promises = [];
promises.push(rp(site1));
promises.push(rp(site2));
promises.push(rp(site3));


Promise.all(promises).then(data => {
    console.log(data);
    // fetch all data from data variable and do calulation 
}, err => {
    console.log('error')
});

Upvotes: 0

Views: 701

Answers (1)

jfriend00
jfriend00

Reputation: 707238

If you want to handle each one individually as the results come in, you can add a .then() and/or .catch() handler for each one:

var promises = [];
promises.push(rp(site1).then(result => {
    // do any partial processing
    return result;
}).catch(err => {
    // decide what to do if error
    throw err;
}));
promises.push(rp(site2).then(result => {
    // do any partial processing
    return result;
}).catch(err => {
    // decide what to do if error
    throw err;
}));
promises.push(rp(site3).then(result => {
    // do any partial processing
    return result;
}).catch(err => {
    // decide what to do if error
    throw err;
}));

Promise.all(promises).then(data => {
    console.log(data);
    // fetch all data from data variable and do calulation 
}, err => {
    console.log('error')
});

Of course, if the processing is identical for each request, then encapsulate that extra functionality in a function that calls rp() and then does the processing and you can then use that function instead of rp(). So, it really depends upon exactly what intermediate processing you want to do.

If the processing is identical, you can create your own wrapper function. For example, supposed you wanted to just keep the processing going, even if error, you could do this:

function rpExtra(args) {
    return rp(args).catch(err => {
        console.log(err);
        return null;   // sentinel resolved value to keep processing going for Promise.all()
    });
}

var promises = [];
promises.push(rpExtra(site1));
promises.push(rpExtra(site2));
promises.push(rpExtra(site3));


Promise.all(promises).then(data => {
    console.log(data);
    // fetch all data from data variable and do calculation
    // skip any `null` values (those had errors) 
}, err => {
    console.log('error')
});

The details of exactly what to do for the error handling here is very situation-specific so I've given you the structure and you'd have to fill in what exactly you want to do. A .catch() handler can either rethrow (to keep the promise rejects and have Promise.all() abort) or it can return a value (to change the promise to resolved and keep Promise.all() processing going).

Upvotes: 1

Related Questions