CODEWITHSUNDEEP
javascriptnode.jsasynchronousaxios
user10365430
user10365430

Reputation:

javascript async help async.timesLimit

I'm trying to scrape a site asynchronously but without doing 500 request at a time but 3 at a time

module.exports.siteRequest = function(n,userInput) {
  payload["numb"]=n
  axios.post(URL, payload, { headers: headers }).then(res => {
      console.log("id",id)
      console.log(res.data)
    })
    .catch(e => {
      console.log("err");
    });
};


async.timesLimit(500,2, function(n, next) {
          siteRequest(n,data)
        });

Upvotes: 1

Views: 157

Answers (1)

CertainPerformance
CertainPerformance

Reputation: 370789

You need to call the .next function in timesLimit in order to signal to the timesLimit iterator that the next item should be requested. For example, the following code will make 10 total requests, with no more than 2 requests at a time:

const makePromise = () => new Promise(res => setTimeout(res, 500)).then(() => {
  console.log('resolving');
});
function siteRequest() {
  console.log('initializing');
  return makePromise();
};


async.timesLimit(10,2, function(n, next) {
  siteRequest().then(next);
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/async/2.6.1/async.min.js"></script>

For your code, make sure to return the Promise returned by axios as well, so you can chain off of it in your timesLimit. You can also use dot notation, which is preferable to bracket notation when possible:

module.exports.siteRequest = function(n,userInput) {
  payload.numb = n
  return axios.post(URL, payload, { headers: headers }).then(res => {
    console.log("id",id)
    console.log(res.data)
  })
    .catch(e => {
    console.log("err");
  });
};
async.timesLimit(500,2, function(n, next) {
  siteRequest(n,data).then(next);
});

Upvotes: 2

Related Questions