CODEWITHSUNDEEP
javascriptnode.jspromise
Return-1
Return-1

Reputation: 2449

Best practice for sequential url requests nodejs

I've got a list of urls i need to request from an API, however in order to avoid causing a lot of load i would ideally like to perform these requests with a gap of x seconds. Once all the requests are completed, certain logic that doesnt matter follows.

There are many ways to go about it, i've implemented a couple.

A) Using a recursive function that goes over an array that holds all the urls and calls itself when each request is done and a timeout has happened

B) Setting timeouts for every request in a loop with incremental delays and returning promises which upon resolution using Promise.all execute the rest of the logic and so on.

These both work. However, what would you say is the recommended way to go about this? This is more of an academic type of question and as im doing this to learn i would rather avoid using a library that abstracts the juice.

Upvotes: 0

Views: 762

Answers (2)

HMR
HMR

Reputation: 39270

You could use something like this to throttle per period.

If you want all urls to be processed even when some fail you could catch the failed ones and pick them out in the result.

Code would look something like this:

const Fail = function(details){this.details=details;};
twoPerSecond = throttlePeriod(2,1000);
urls = ["http://url1","http://url2",..."http://url100"];
Promise.all(//even though a 100 promises are created only 2 per second will be started
  urls.map(
    (url)=>
      //pass fetch function to twoPerSecond, twoPerSecond will return a promise
      //  immediately but will not start fetch untill there is an available timeslot
      twoPerSecond(fetch)(url)
      .catch(e=>new Fail([e,url]))
  )
)
.then(
  results=>{
    const failed = results.map(result=>(result&&result.constuctor)===Fail);
    const succeeded = results.map(result=>(result&&result.constuctor)!==Fail);
  }
)

Upvotes: 0

Akxe
Akxe

Reputation: 11495

Your solutions are almost identical. Thought I would choose a bit different approach. I would make initial promise and sleep promise function, then I would chain them together. 

function sleep(time){
    return new Promise(resolve => setTimeout(resolve, ms));
}

ApiCall()
.then(sleep(1000))
.then(nextApiCall())...

Or more modular version

var promise = Promise.resolve()
myApiCalls.forEach(call => {
    promise = promise.then(call).then(() => sleep(1000))
})

In the end, go with what you understand, what make you most sense and what you will understand in month. The one that you can read best is you preferred solution, performance won’t matter here.

Upvotes: 1

Related Questions