CODEWITHSUNDEEP
javascriptnode.js
Sergei Basharov
Sergei Basharov

Reputation: 53850

How to control number of simultaneously running async http get requests?

I have a list of generated on-the-fly urls (about 30-40) which I want to get via http.get() and save as files.

I want to make sure that not all of them run at once because I don't want to overwhelm nor server neither my utility client app.

How do I limit the number of these requests running?

Upvotes: 1

Views: 216

Answers (1)

ralphtheninja
ralphtheninja

Reputation: 133008

With the async module you can use for example parallelLimit(). You can specify maximum simultaneous requests.

Example with max three requests at a time (note that you need to tweak it):

const http = require('http')
const async = require('async')

const urls = [ /* array of urls */ ]
const fns = urls.map(url => {
  return cb => {
    http.get(url, res => {
      // handle res, call cb() when done
      cb()
    })
  }
})

async.parallelLimit(fns, 3, err => {})

For reference: https://caolan.github.io/async/docs.html#parallelLimit

Upvotes: 1

Related Questions