CODEWITHSUNDEEP
javascriptnode.jsasynchronousaxios
jsPlayer
jsPlayer

Reputation: 1245

Node js how to run axios.get every 2 seconds?

I am kinda of a newbie to node js, Here is what i am trying to do: i am looping through a json file full of links of our website via the map function (around 3000 links), inside the loop i am doing a axios get for each link and getting the response status code(will do other things in the future). But i want to run the axios get only like every 2 seconds or 5 seconds otherwise i am overwhelming the webserver. I am trying to input async await but it's still too fast and server is taking a hit (i am technically DDos-ing my own website). I put a SetTimeout around the axios but that doesn't seem like it worked, since in the console results are printing way too fast. so the question is, how do i make each axios.get request wait every 2 seconds before running in the map loop?.

var axios = require('axios');
const fs = require('fs');
var statusCheck = 0;

var main = [];
let rawdata = fs.readFileSync('C:/Users/jay/Documents/crawl/filtered2.json');
let jsonParsed = JSON.parse(rawdata);

jsonParsed.map(async(line) => {

    var encodeLink = encodeURI(line.link);
    const response = await axios.get(encodeLink).catch((err) => {
        var Status_ErrorsCatchaxios = {
            "status Code": err.response.status ? err.response.status : "No status code available",
            "Page title:": $('title').text() ? $('title').text() : 'No title avaialble',
            "Original Link": encodeLink ? encodeLink : "No Original Link Available",
            "errorCode": err
        }
        main.push(Status_ErrorsCatchaxios)
    })
    try {
        console.log(response.status)

        statusCheck = statusCheck + 1;
        console.log("Link: ", statusCheck)
    } catch (error) {
        console.log(error)
    }
})

Upvotes: 8

Views: 11696

Answers (6)

user10397650
user10397650

Reputation: 11

You can use

var i=0;
jsonParsed.map(async(line) => {
i++
setTimeout(async() => {

},i*2000)
}

Upvotes: 0

Guilherme Sehn
Guilherme Sehn

Reputation: 6787

The [].map function doesn't wait for your items to resolve, so your code is currently dispatching all the requests (as you said, around 3000) in parallel.

You can use for...of instead to only run one request at a time. For example:

async function makeRequests (lines) {
  for (const line of lines) {
    const encodedLink = encodeURI(line.link)
    const response = await axios.get(encodedLink)
    // ...your response handling code here...
  }
}

makeRequests(jsonParsed)

If you want to wait for 2s between each request, you can add this line of code inside your for...of loop:

await new Promise(resolve => setTimeout(resolve, 2000))

Better solution

The solution above works, but I assume your webserver can probably take more than one request at a time, so maybe the ideal scenario would be to limit your code to make only up to N requests in parallel at a given time. This way you don't flood your server but you're able to get your results faster than just doing one request at a time.

The bluebird NPM module allows you to do that with their Promise.map function.

This function receives your list of items as the first argument, a function that executes something and returns a promise for each item as the second argument, and an object with a concurrency key describing how many items you want to allow to be handled in parallel as the third argument.

Here's how it could work:

const bluebird = require('bluebird')

async function makeRequests (lines) {
  await bluebird.map(
    lines,
    async (line) => {
      const encodedLink = encodeURI(line.link)
      const response = await axios.get(encodedLink)
      // ...your response handling code here...
    },
    { concurrency: 3 }
  )
}

makeRequests(jsonParsed)

Upvotes: 7

Amir akbari
Amir akbari

Reputation: 62

you can use setTimeout function for running codes every 2 second!

setTimeout(async() => {
 // await postRequest()
},2000)

Upvotes: -1

Ravi Shankar Bharti
Ravi Shankar Bharti

Reputation: 9268

The reason why timeout wont work in a loop is because it will fire all the requests/functions all at once after the timeout delay.

The idea is to put delay in each iteration and only after delay start the next iteration.

you can run a self invoking function which calls itself after the delay. So to run the function every 2 seconds, you can try this:

let jsonParsed = JSON.parse(rawdata);
let len = jsonParsed.length;

(function requestLoop (i) {          
    setTimeout(function () {   
        let line = jsonParsed[len-i]
        var encodeLink = encodeURI(line.link);
        const response = await axios.get(encodeLink).catch((err) => {
            var Status_ErrorsCatchaxios = {
                "status Code": err.response.status ? err.response.status : "No status code available",
                "Page title:": $('title').text() ? $('title').text() : 'No title avaialble',
                "Original Link": encodeLink ? encodeLink : "No Original Link Available",
                "errorCode": err
            }
            main.push(Status_ErrorsCatchaxios)
        })
        try {
            console.log(response.status)

            statusCheck = statusCheck + 1;
            console.log("Link: ", statusCheck)
        } catch (error) {
            console.log(error)
        }                        
        let jsonParsed = JSON.parse(rawdata);

        if (--i) requestLoop(i);     
   }, 2000)
})(len);

Upvotes: 1

Aritra Chakraborty
Aritra Chakraborty

Reputation: 12542

You are hitting all at once because .map,.forEach,.reduce etc doesn't wait for the Promise to resolve. Use Simple For loop, it will wait for each promise to resolve or reject.

for(let i=0;i<jsonParsed.length;i++) {
    var encodeLink = encodeURI(line.link);
    const response = await axios.get(encodeLink).catch(...)
    try {
        ....
    } catch (error) {
        ...
    }
})

Why it doesn't work? If we imitate the forEach loop it will be something like,

function forEach(arr, cb){
    for(let i=0;i<arr.length;i++){
        cb(arr[i], i, cb);
    }
}

So you see it doesn't await the cb.

Upvotes: 1

Nino Filiu
Nino Filiu

Reputation: 18493

Ditch the map, replace with a for ... of, await a promise that takes 2s to resolve, wrap everything inside an async IIFE for the await to be legal.

// dummy data
const fakeJson = new Array(5).fill();
const fakeRequest = () => console.log(`request at ${new Date().toUTCString()}`);

// iteration with 2s in between
(async () => {
  for (let line of fakeJson) {
    await new Promise(r => setTimeout(r, 2000));
    fakeRequest();
  }
})()

You can also use more classically use setInterval but HTTP requests are asynchronous, so might as well start with a structure that handles well async and loops.

Upvotes: 2

Related Questions