CODEWITHSUNDEEP
node.jsrequest
MrJalapeno
MrJalapeno

Reputation: 1662

Making multiple requests in node js

So I have this overhanging fear this is a question asked many times before but I've searched and every question seem to be tailored to different specific circumstances. If you see it as a duplicate, please flag it and point me in the right direction.

A general minimal example:
I have an array data and want to make an API-call for every element in data, adding things to each element. Then I want to get the processed data back.

My first attempt (in pseudo:ish code):

// function that recursively makes the calls
makeCalls(data,i,resolve){
    // base case, we're done
    if(i === data.length){
        return resolve(data);
    }
    const options = {
        url: "http://theapi.com/?name="+data[i].name
    };
    request.post(options,function(error,response,body){
        data[i].item = body.item;
        i++;
        return makeCalls(data,i,resolve);
    }
}

// later on in my code I call the function
new Promise(resolve => makeCalls(data,0,resolve))
    .then(returnedData => console.log("done!"))

Now that solution worked (although it took 264s for 680 calls), but since the calls weren't dependent on each other I thought I could speed it up so I made a loop instead of recursion:

// function that iteratively makes the calls
makeCalls(data,resolve){
    let j = 0
    const options = {
        url: "http://theapi.com/?name="+data[i].name
    };
    for(let i = 0; i < data.length; i++){
        request.post(options,function(error,response,body){
            data[i].item = body.item;
            j++;
            if(j === data.length)
                return resolve(data);
        }
    }
}

Now 680 calls takes just 56 seconds. Perhaps that's perfectly normal but since I've been winging my code pretty much I'd really like to know if there's an "idiomatic" way of doing this in node? I mean should I really be making the Promise like that and passing the resolve-object into the function? Is there a "proper" way of making a bunch of API-calls at once?

If anyone is curious the data is an array of tracks from the Spotify-API which unfortunately only provide genres on artist level. I need genres on track-level and therefore turn to the LastFM-API which has an endpoint giving me the top-tags of a track. The only issue is that I need to make this call for every single track :(

Upvotes: 0

Views: 1114

Answers (1)

Gonzalo.-
Gonzalo.-

Reputation: 12682

I would use a wrapper on request to return a promise. See the list

with that in mind, I would simplify your code using a map function to make every request and update all items, and Promise.all() to wait for all your promises

Using request-promise for the example

const options = {
        url: "http://theapi.com/?name=" // asumimng you're using an object because there are other options
    };

let promises = data
      .map(element => rp({ ...options, url: options.url + element.name })
                .then(response => element.item = response.item)
      );
return Promise.all(promises).then(() => console.log('done!'));

Any way, what you fixed here mostly is making the code cleaner and declarative. Probably the time is taking you is because all those requests are slow per se. Does the API you're using offer you a chance to send multiple ids in one request or something similar? You might want to do this in one request maybe

Upvotes: 1

Related Questions