CODEWITHSUNDEEP
javascriptnode.jsqueueworker
user1159470
user1159470

Reputation: 299

How to create and manage worker processes in NodeJS?

For example, say the user requests some audio file to be processed, then of course nodejs can't do the intense processing, so it should offload it to a worker process.

These workers probably need to be able to pub/sub to events, respawn when they die, and the queue should be able to load balance, maintain a cache, and keep alive. I've seen 0MQ, and others like it, but I'm not sure how I would go about integrating it into a web app...

What is industry standard way to create and manage these worker processes? And what are the tools used?

Edit: One more thing: say the audio processing takes a long time, and the request times out. Is there a way to get around that other than increasing the timeout?

Edit 2: By workers, I mean like the Heroku worker dynos - how do they work?

Upvotes: 4

Views: 1059

Answers (1)

mtsr
mtsr

Reputation: 3152

Personally I'd probably do something like add the tasks to a redis database. Then another process (probably written in a different language if it's very processing heavy) is subscribed to that key in the redis database and starts tasks that get added to it. It could even manage some worker threads itself.

For the timeout you could use one request to only start the task and another request to get the results (with longpolling if the result isn't available yet, or even better server-push through websockets, for example).

Upvotes: 1

Related Questions