CODEWITHSUNDEEP
javascriptnode.js
S. Schenk
S. Schenk

Reputation: 2150

Delaying a Promise.all sequence

I'm fetching data from an external API and I get the following error on some of the endpoints:

reason: getaddrinfo EAI_AGAIN [the url]:443

I can only assume that the owner of the API has a rate limiter of some sort and since all the fetching is done at once some URLs get blocked.

So I'm trying to put a delay between each fetch, but with no luck. Here is my try:

class someClass {
  ...
  dealyedFetch(url) {
    return new Promise ((resolve, reject) => {
      setTimeout(() => {
        fetch(url, (err, res) => { 
          if (err) return reject(err)
          return resolve(res)
        })
      }, 5000)    
    })
  }

  fetchUrls() {
    return Promise.all(
      this.urlList.map(url => {
        return this.dealyedFetch(url)
        .then(/* macht krieg */)
        .catch(err => ({err, url}))
      })
    )
  }    
}

the above doesn't work and it seems to fire all the fetching at once. Please illuminate me.

Upvotes: 3

Views: 2685

Answers (2)

intentionally-left-nil
intentionally-left-nil

Reputation: 8274

For your urlList, you are making a series of promises at the same time. Each of these promises waits 5 seconds, and then they all execute simultaneously (5 seconds later).

See this for the canonical SO post on running promises in serial: Resolve promises one after another (i.e. in sequence)?

Lifted from that post:

function runSerial(tasks) {
  var result = Promise.resolve();
  tasks.forEach(task => {
    result = result.then(() => task());
  });
  return result;
}

and then you would consume it by replacing your promise.all with runSerial. You can further customize the logic by releasing like 5 url's at once, and then running the batches in serial, but this should get you started.

Upvotes: 0

T.J. Crowder
T.J. Crowder

Reputation: 1074168

You're delaying them all at once, so they all wait 5000ms, then fire together.

If you want to string them out, you need to make their delays progressively longer. You could do that by using the index in the map to multiple a delay of, say, 500ms:

delayedFetch(url, delay) {                   // ***
  return new Promise ((resolve, reject) => {
    setTimeout(() => {
      fetch(url, (err, res) => { 
        if (err) return reject(err)
        return resolve(res)
      })
    }, delay)                                // ***
  })
}
fetchUrls() {
  return Promise.all(
    this.urlList.map((url, i) => {           // ***
      return this.delayedFetch(url, i * 500) // ***
      .then(/* macht krieg */)
      .catch(err => ({err, url}))
    })
  )
}

The first request won't be delayed at all, the second will be 1 * 500 = 500ms, the third will be 2 * 500 = 1000ms, etc.

Side note: In the above I fixed a typo in the name delayedFetch (it was dealyedFetch in the question, a and l were reversed).

Side note 2: The above addresses what you actually asked, but you might consider doing them one at a time, in which case this question's answers show you how to do that. Or perhaps a concurrency limit on the outstanding ones, in which case this question's answers would help. Just a fixed delay may not be your best solution.

Upvotes: 4

Related Questions