CODEWITHSUNDEEP
node.jsasynchronouspromisebluebird
CoffeePeddlerIntern
CoffeePeddlerIntern

Reputation: 679

Delaying promises: .delay() doesn't seem to work

This is an extended question from Throttling requests to 3rd part Apis with Node . You can see my full code in that post if you want to see the sub functions and a little backstory of what im trying to do.

What i need to do is delay api request to an api. I am hitting the server max per second. I am trying to use Promise.delay but it doesnt seem to hold anything. This is how im using it.

function getAllRegions(req, res){
    getAllRegionsHrefs().then(function (hrefs){
        var chunks = _.chunk(hrefs, 25);
        return Promise.map(chunks, function(chunk) {
            return Promise.map(chunk, getRegion).then(function (getRegionResults){
                for(var item in getRegionResults) {
                    Promise.map(getRegionResults[item].constellations, getConstellationInfo).then(function (constellationInfo) {
                        var chunks = _.chunk(constellationInfo, 150);
                        return Promise.map(chunks, function (chunk) {
                            return Promise.map(chunk, getSystem).delay(20000);
                        })
                    }).delay(20000);
                }
            }).delay(200000);
        });
    });
}

Now this will make just about 9k api requests and it builds a universe based off regions, constellations and systems. Is there a better way of throttling the nested requests? At each nest, the number of calls exponentially goes up.

Update

Still doesnt seem to want to wait. Did I mess up the nesting again? I added 1 more function in there that was needed for the flow. Also the chunking seemed to wrap a double array so I took it out. 

function getAllRegions(req, res) {
    getAllRegionsHrefs().then(function (hrefs) {
        return Promise.mapSeries(hrefs, function (href) {
            getRegion(href).then(function (regions) {
               return Promise.mapSeries(regions.constellations, function (constellation) {
                    getConstellationInfo(constellation).then(function (constellationInfo) {
                        return Promise.map(constellationInfo, getSystem).delay(15000);
                    });
                });
            });
        });
    });
}

Error (per request):

 Unhandled rejection RequestError: Error: connect ETIMEDOUT

Upvotes: 0

Views: 422

Answers (1)

DarkKnight
DarkKnight

Reputation: 5934

Promise.map starts all requests in parallel, so your code runs like:

  • requests -> delay
  • requests -> delay
  • requests -> delay
  • ....

But the desired behavior should be sequential:

  • requests -> delay -> requests -> delay -> requests -> delay ...

each chunk of requests should wait for delay of previous chunk.

You could achieve this by chaining promises:

startRequests(chunks[0])
.then(function() {
  return startRequests(chunks[1])
}).then(function() {
  return startRequests(chunks[2])
}) // ...

or use mapSeries

function getAllRegions(req, res){
  getAllRegionsHrefs().then(function(hrefs) {
    return Promise.mapSeries(hrefs, function(href) {
      getRegion(href).then(function(region) {
        var chunks = _.chunk(region.constellations, 150);
        return Promise.mapSeries(chunks, function (chunk) {
          return Promise.map(chunk, getSystem).delay(10000);
        });
      });
    });
  });
}

only innermost requests are chunked, others are started one by one.

edit: as Bergi said, returning result promise is required for mapSeries (and other promise utilities) to work. The first promise can also be returned for handling error or chaining more promises.

function getAllRegions(req, res){
  return getAllRegionsHrefs().then(function(hrefs) {

//...

getAllRegions().then(systems => {
  //next step
}).catch(err => {
  //handle errors
})

Upvotes: 1

Related Questions