CODEWITHSUNDEEP
node.js
wong2
wong2

Reputation: 35700

Does Node.js need a job queue?

Say I have a express service which sends email:

app.post('/send', function(req, res) {
  sendEmailAsync(req.body).catch(console.error)
  res.send('ok')
})

this works.

I'd like to know what's the advantage of introducing a job queue here? like Kue.

Upvotes: 11

Views: 16380

Answers (2)

jfriend00
jfriend00

Reputation: 707148

Does Node.js need a job queue?

Not generically.

A job queue is to solve a specific problem, usually with more to do than a single node.js process can handle at once so you "queue" up things to do and may even dole them out to other processes to handle.

You may even have priorities for different types of jobs or want to control the rate at which jobs are executed (suppose you have a rate limit cap you have to remain below on some external server or just don't want to overwhelm some other server). One can also use nodejs clustering to increase the amount of tasks that your node server can handle. So, a queue is about controlling the execution of some CPU or resource intensive task when you have more of it to do than your server can easily execute at once. A queue gives you control over the flow of execution.

I don't see any reason for the code you show to use a job queue unless you were doing a lot of these all at once.

The specific https://github.com/OptimalBits/bull library or Kue library you mention lists these features on its NPM page:

  • Delayed jobs
  • Distribution of parallel work load
  • Job event and progress pubsub
  • Job TTL
  • Optional retries with backoff
  • Graceful workers shutdown
  • Full-text search capabilities
  • RESTful JSON API
  • Rich integrated UI
  • Infinite scrolling
  • UI progress indication
  • Job specific logging

So, I think it goes without saying that you'd add a queue if you needed some specific queuing features and you'd use the Kue library if it had the best set of features for your particular problem.

In case it matters, your code is sending res.send("ok") before it finishes with the async tasks and before you know if it succeeded or not. Sometimes there are reasons for doing that, but sometimes you want to communicate back whether the operation was successful or not (which you are not doing).

Upvotes: 15

samanime
samanime

Reputation: 26527

Basically, the point of a queue would simply be to give you more control over their execution.

This could be for things like throttling how many you send, giving priority to other actions first, evening out the flow (i.e., if 10000 get sent at the same time, you don't try to send all 10000 at the same time and kill your server).

What exactly you use your queue for, and whether it would be of any benefit, depends on your actual situation and use cases. At the end of the day, it's just about controlling the flow.

Upvotes: 6

Related Questions