CODEWITHSUNDEEP
node.jspromiserequest-promise
Ridhwaan Shakeel
Ridhwaan Shakeel

Reputation: 1043

Rate limiting request-promise in for loop

I'm trying to to limit the rate at which I'm firing GET requests to an outside API using promises but I'm having difficulty getting it working. In my scenario, I'm using 'request-promise' module and I need to send GET requests for 175 items from an API (one request for each item ID). The API has a rate limit of 40 requests per 10 seconds so my throttle needs to be 250ms per request. I am trying to send a request inside loop for each item ID, like:

files.forEach(function (file, i) {
    console.log("The item ID is " + file.match(re)[1]);
    client.send(new APIClient.requests.getItem(file.match(re)[1]))
    .then((item) => {
      ...
    })
    .catch((error) => {
      console.error(error);
      // Use fallback
    });
    ...

Here is a snippet of my API client return a request promise (rp) with a 250ms timeout and no callback:

const rp = require('request-promise');
const rp_errors = require('request-promise/errors');
...
send(request, callback) {
...
    return rp(options)
               .then(this._parseResponse)
               .then((response)=> {
                  return new Promise( (resolve) => setTimeout(function(){
                    if (callback) { return callback(null, response); }
                    return resolve(response);
                  }, 250));
                })
                .catch(rp_errors.StatusCodeError,((error) => {
                    throw new errors.ResponseError(request, error.statusCode, error.message);
                  }
                ))
                .catch(rp_errors.RequestError,((error) => {
                    if(error.cause.code === 'ETIMEDOUT' || error.cause.code === 'ESOCKETTIMEDOUT')
                      throw new errors.TimeoutError(request, error);
                    throw error;
                  }
                ))
                .catch((error) => {
                  if (callback) {return callback(error)};
                  throw error;
                });
}

The Async doesnt work and it returns this stack trace of "429 request limit exceeded"

{ ResponseError: 429 - {"status_code":25,"status_message":"Your request count (175) is over the allowed limit of 40."}
[0]     at rp.then.then.catch (/mnt/c/Users/ridhwaan/Source/homehost/lib/api-client.js:52:19)
[0]     at tryCatcher (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/util.js:16:23)
[0]     at /mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/catch_filter.js:17:41
[0]     at tryCatcher (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/util.js:16:23)
[0]     at Promise._settlePromiseFromHandler (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:512:31)
[0]     at Promise._settlePromise (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:569:18)
[0]     at Promise._settlePromise0 (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:614:10)
[0]     at Promise._settlePromises (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:689:18)
[0]     at Async._drainQueue (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/async.js:133:16)
[0]     at Async._drainQueues (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/async.js:143:10)
[0]     at Immediate.Async.drainQueues [as _onImmediate] (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/async.js:17:14)
[0]     at runCallback (timers.js:756:18)
[0]     at tryOnImmediate (timers.js:717:5)
[0]     at processImmediate [as _immediateCallback] (timers.js:697:5)
[0]   name: 'ResponseError',
[0]   request:
[0]    Movie {
[0]      method: 'GET',
[0]      path: '/movie/24428',
[0]      timeout: 10000,
[0]      ensureHttps: false,
[0]      external_id: '24428' },
[0]   statusCode: 429 }

Upvotes: 1

Views: 4190

Answers (1)

Kroney
Kroney

Reputation: 73

So the main issue is that array.forEach is a synchronous function and will not wait for client.send to finish. A solution would be to use bluebird.mapSeries (http://bluebirdjs.com/docs/api/promise.mapseries.html) to map through an array and wait for each iteration to finish. Also don't forget to return the send-promise, so the mapSeries-function will know when it has resolved, so it knows when to start the next iteration:

bluebird.mapSeries(files, function(file){
  return send(...)
}

Last a suggestion would be to replace the whole .then(... setTimeout...) part with just .delay(250). Request-Promise already uses bluebird promises so you can make use of their convenience functions http://bluebirdjs.com/docs/api/delay.html. Delay will automatically resolve the value from the previous promise

return rp(options)
   .then(this._parseResponse)
   .delay(250)
   .catch(...)

Upvotes: 1

Related Questions