HenryM
HenryM

Reputation: 5793

NodeJS Promise, how to spread/slow http.get calls

I am new to NodeJS and to Promise functionality so please be polite if this is an ignorant question.

I'm trying first read a database of records and then check the links actually work (looking for a 200 response). For my current test data this should will always return a 200 response. I'm getting 302 (too many requests) response and then the development server crashes. I need to slow down how I send through requests to the database but I cannot work out how to do it. It seems to me the promise simply sends everything as soon as it is resolved.

I have tried building in time delays in the then section, but to no avail. Here's the code:

var http404Promise = new Promise(function(resolve, reject) {
        var linkArray = new Array()

        db.sequelize.query(photoQuery, {
            replacements: queryParams
        }).spread(function(photoLinks) {
            photoLinks.forEach(function(obj) {
                var siteLink = hostname + 'photo/' + obj.img_id
                linkArray.push(siteLink)

                //console.log(siteLink);
            });

            resolve(linkArray);
        });
    });

http404Promise.then(function(linkArray) {
    linkArray.forEach(function(element) {
        console.log(element);
        http.get(element, function(res) {
            console.log(element);
            console.log("statusCode: ", res.statusCode); // <======= Here's the status code
        }).on('error', function(e) {
            console.log(element);
            console.error(e);
        })
    })    
});

Upvotes: 3

Views: 718

Answers (4)

Nishant Dixit
Nishant Dixit

Reputation: 5522

You can use async module for this

https://github.com/caolan/async

or if you wish to go with current code then you need to change the follow

linkArray = [];
var counter = 0;
http404Promise.then(function(responseData) {
  linkArray  = responseData;
  if (linkArray.length > 0) {
    requestHandler(linkArray.pop());
  }
});

function requestHandler(data) {
  http
    .get(data, function(res) {
      counter++;
      if (counter == linkArray.length) {
        console.log("finished");
      } else {
        reuqestCheckPromiss(linkArray.pop());
      }
    })
    .on("error", function(e) {
      counter++;
      if (counter == linkArray.length) {
        console.log("finished");
      } else {
        reuqestCheckPromiss(linkArray.pop());
      }
    });
}

Upvotes: 0

lukas-reineke
lukas-reineke

Reputation: 3322

The reason a normal timeout in the forEach doesn't work, is that the forEach does not wait for the timeout to finish. So every request waits simultaneously, and you don't get the staggering you want.
You can however use the index of each element to calculate the timeout.

linkArray.forEach(function(element, index) {
    setTimeout(function() {
        console.log(element);
        http.get(element, function(res) {
            console.log(element);
            console.log("statusCode: ", res.statusCode); // <======= Here's the status code
        }).on('error', function(e) {
            console.log(element);
            console.error(e);
        });
    }, index * 500);
});

Upvotes: 1

Bartłomiej Gładys
Bartłomiej Gładys

Reputation: 4615

You can use reduce function to create pipe of delayed requests

    const timeout = 500;
    const linkArray = [1,2,3,4]
    
    const httpFunc = el => {
      // your http function here
      console.log('request num ', el)
    }
    
    
    const func = el => new Promise(resolve => {
      setTimeout(() => {
        resolve(httpFunc(el));
      }, timeout)
    })
    
    linkArray.reduce((acc, el) => acc.then(() => func(el)), Promise.resolve())

Upvotes: 0

rahulroy9202
rahulroy9202

Reputation: 2848

linkArray.forEach(function(element) {
        .....
    })

Here instead of doing forEach you can use Async eachLimit to do a few of the requests at a time. You can also write the code such that the next request is done after the previous one completes or use Async eachSeries but with eachLimit you have a bit of parallelism.

Upvotes: 0

Related Questions