CODEWITHSUNDEEP
node.jsapi
pilotGuy
pilotGuy

Reputation: 111

Rate limit API requests in node.js

First off, I'm new here, so please...be gentle...I've been teaching myself node.js over the last few months, mostly with the desire to scrape a bunch of data from the FlightAware website API.

I am trying to request from their site a list of flights for aircraft, here is what I have so far. 

var aircraft = [array,of,aircraft,tail,numbers]   

for (i=0; i < aircraft.length; i++) {

    faGetFlight(aircraft[i],function doneLookup(data) {

        dbUpdateFlight(collectionName, data)
    })
}

This code works, but as soon as there is more than 10 aircraft in the list, it fails, because is sending more than 10 API requests in a minute. What are some easy/straightforward ways to slow this down a little. I would like to send about 50-60 API requests total each time this runs, so I need it spaced over 5-6 minutes. The faGetFlight() function uses the 'request' module. I've tried the request-rate-limit module instead of the request module, but I can't seem to make it work. I don't think the authorization works properly with the request-rate-limiter module. Getting an error about anonymous user. For what it's work, it works with just the request module instead, but I run into the rate limit issues.

Here is my faGetFlight() code. 

var RateLimiter = require('request-rate-limiter');
const REQS_PER_MIN = 10; // that's 25 per second
var limiter = new RateLimiter(REQS_PER_MIN);

//API Variables //
var apiUrl = 'url'
var apiEndpoint = 'endpoint'
var apiAuth = 'apikey'
var apiExtData = 0
var apiHowMany = 15         //Number of results to request.                                               

var options = { method: 'GET',
    url: apiUrl + apiEndpoint,
    qs: { ident: acIdent
        },

        headers: 
    { 
        Authorization: apiAuth }                  
    };

        limiter.request(options, function doneDownload(error, response, body) {         
            if (error) throw new error(error);
            callback(body)
        });

}

Sorry if this isn't clear...it's my first post!

Upvotes: 3

Views: 2964

Answers (1)

Azami
Azami

Reputation: 2161

You can do a naive implementation using functions and a simple setTimeout.

See:

var aircrafts = [array,of,aircraft,tail,numbers]; 
var REQS_PER_MIN = 10;
var MAX_AMOUNT_REQUESTS = 100;
var timeout = (1 / REQS_PER_MIN) * 60 * 1000;

processAircraft(0);

function processAircraft(index){
    if(index >= MAX_AMOUNT_REQUESTS)
        return console.log("All done!");
    //On start of function, schedule next processing in "timeout" ms
    setTimeout(function(){
        processAircraft(index+1);
    }, timeout);

    faGetFlight(aircrafts[index], function doneLookup(data) {
        dbUpdateFlight(collectionName, data)
    })
}

Upvotes: 0

Related Questions