CODEWITHSUNDEEP
node.js
dzm
dzm

Reputation: 23534

node.js worker farm and queue management

I'm looking for a solution to create a "worker farm" using node.js. Basically we have an app in node and we need to send off "jobs" to be run across n number of worker servers. For example, let's say we have 5 servers that all run certain jobs, the jobs need to be distributed or 'queued' until a worker has CPU available to process the jobs.

One way to do this would be to have a worker server that is run on every separate machine. Each worker would pull from a queue based on it's CPU utilization or queue availability. The application itself, would add items to a queue (probably handled by Redis). There would be no direct communication between the individual worker servers and the application itself. One problem I could see with this is if multiple workers grab the same queue at the same time. The other method would be to somehow communicate with the worker servers from the application, which would get worker with the least resources and 'assign' the job to that particular worker or queue it up.

Does anyone know of a good solution for handling this?

Thank you!

Upvotes: 1

Views: 1556

Answers (1)

Dan Kohn
Dan Kohn

Reputation: 34327

I recommend kue, which runs on top of Redis. It gives you atomic queue operations, and each worker can get the next task from the queue. Take a look at resque for a more full-featured version of the same.

Upvotes: 3

Related Questions