CODEWITHSUNDEEP
javascriptnode.jsajaxaxiosthrottling
Alex Karalanian
Alex Karalanian

Reputation: 81

Throttling API AJAX requests in Node (getting rate limited) - Using AXIOS for AJAX calls...

I am getting rate limited making too many requests per second to an API.

I am importing contacts into my database and mapping the data to my DB's schema.

The API initially returns its contact 'groups' (ie categories) as an array of links which need to all have separate API calls made in order to retrieve the actual value.

I must iterate over every contact AND make separate API calls for every link in the array associated with each contact. I quickly hit my rate limit (max 10 per second according to API docs) and get a bunch of incomplete entries and errors as a result.

I am using Axios library to make the requests in Node.

My code is below - any help solving this dilemma would be greatly appreciated - thank you!

        .then(authUser => {

            // RETURN A LIST OF CONTACTS FROM API

            axios.get('url', {
              headers: {
                Authorization: credentials
              }
            })
            .then(response => {

              // MAP OVER EACH CONTACT
              response.data.map( contact => {

                let groups = contact.groupLinks.map(group => {

                    // MAP OVER EACH LINK IN EACH CONTACTS GROUP ARRAY AND MAKE AJAX REQUEST. 
                    return axios.get(`${group.url}/?alt=json`, {
                      headers: {
                        Authorization: 'credentials'
                      }
                    })
                    .then(result => {
                      return result.data
                    })
                    .catch(err => {
                      console.error('FETCH GROUP MEMBERSHIP FAILURE', err.response.data.error)
                      next(err)
                    })
                  })

                return Promise.all(groups)
                .then( response => {
                  return [contact, response]
                })

Upvotes: 3

Views: 1120

Answers (2)

Alex Karalanian
Alex Karalanian

Reputation: 81

Well, I ended up doing something completely different as this proved to be too complicated and inefficient. I couldn't get it to work.

The API allows you to download a complete list of all the user's groups. I created a new DB table called 'ContactGroups'

For Contacts the API returns an object which includes an array of groups IDs.

To display the groups I make a separate POST request to an API endpoint call '/getgroups' with the groupIds array in payload and make the following db query: 

ContactGroups.findAll({
      where: {
        groupId: req.body.groups // [this is an array of ids]
      }
    })

This returns an array of the matched groups.

Upvotes: 0

Russell
Russell

Reputation: 496

I'm not familiar with AXIOS so I'm not sure this is possible. When you use the array/object map to have each group call it's url how would an external variable be bound? Could you do something like this? 

// MAP OVER EACH CONTACT
response.data.map( contact => {
/*right here set an interval
*/
var interval = 0;
let groups = contact.groupLinks.map(group => {
    //increment interval
    interval += 150;

    // MAP OVER EACH LINK IN EACH CONTACTS GROUP ARRAY AND MAKE AJAX REQUEST. 
    setInterval(function()
      {
          return axios.get(`${group.url}/?alt=json`, {
          headers: {
            Authorization: 'credentials'
          }
        })
        .then(result => {
          return result.data
        })
        .catch(err => {
          console.error('FETCH GROUP MEMBERSHIP FAILURE', err.response.data.error)
          next(err)
        })
      }, interval);
  })

Otherwise I would move the processing of the group response to a recursive function that accepts the group returned from the API, makes the necessary calls, pop the group just called, then calls itself on a 150ms delay.

Upvotes: 1

Related Questions