CODEWITHSUNDEEP
node.jsmongodbredisqueuejobs
IyadAssaf
IyadAssaf

Reputation: 308

Nodejs: Async job queue processing

I am working in nodejs with express for a web app that communicates with mongodb frequently. Currently I running production with my own job queue system that only begins processing a job once the previous job has been completed (an approach that seems to be taken by kue).

To me, this seems wildly inefficient, I was hoping for a more asynchronous job queue, so I am looking for some advice as to how other nodejs developers queue their jobs and structure their processing.

One of my ideas is to process any jobs that are received immediately and return the resulting data in the order of addition.

Also to be considered: currently each user has their own independent job queue instance, is this normal practice? Is there any reason why this should not be the case? (ie, all users send jobs to one universal queue?)

Any comments/advice are appreciated.

Upvotes: 1

Views: 4986

Answers (1)

CFrei
CFrei

Reputation: 3627

Why do you build your own queue system? You did quite a lot of work to serialize a async queue with addLocalJob.

Why don't you just do something like

on('request', function(req, res) { queryDB(parameter, function(result) { res.send(result) })

? Full parallel access, no throttle, no (async) waiting. If you really want to do it "by hand" in your own code, why not execute the first n elements of your trafficQueue instead of only the first?

If you want to throttle the DB - two ways:

  • use a library like async and the function parallelLimit
  • connect to your mongodb with http(s) and use the node-build-in http.globalAgent.maxSockets.

Hope this helps.

Upvotes: 0

Related Questions