CODEWITHSUNDEEP
javascriptnode.jsexpress
user6017908
user6017908

Reputation:

How to run simultaneous Node child processes

TL;DR: I have an endpoint on an Express server that runs some cpu-bound logic in a child_process. The problem is that if the server gets more than one request for that endpoint it won't run both requests simultaneously- it queues them up and runs them one-at-a-time. Is there a way to use Node child_process so that my server will perform multiple child processes simultaneously?

Long-Version: The major downfall of Node is that it is single-threaded and a logic-heavy (cpu-bound) request can make the server stop dead in its tracks so that it can't take anymore requests until that logic is finished running. I thought that I could work around this using child_process, which is working great in freeing up my server to take other requests. BUT- it will only execute child_processes one at a time, creating a queue that can get pretty backed-up. I also have a Node cluster setup so that my server is split into 8 separate "virtual servers" (8-core machine), so I guess I can technically run 8 of these child processes at once, but I want to be able to handle more traffic than that. Looking for a solution that will still allow me to use Node and Express, please only suggest using different technologies if you are absolutely sure this can't be efficiently done in my current environment. Thanks in advance for the help!

Endpoint:

app.get('/cpu-exec-file', function(req, res) {
    child_process.execFile('node', ['./blocking_tasks/mathCruncher.js'], {timeout:30000}, function(err, stdout, stderr) {
        var data = JSON.parse(stdout);
        res.send(data);
    })
});

mathCruncher.js:

var obj = {}

function myLoop (i) {
    setTimeout(function () {
        obj[i] = Math.random() * 100;
        if (--i) {
            myLoop(i);
        } else {
            string = JSON.stringify(obj);
            console.log(string); // goes to stdout.
        }
    }, 1000)
};

myLoop(10);

Upvotes: 0

Views: 1098

Answers (1)

Derick Bailey
Derick Bailey

Reputation: 72858

Is there a way to use Node child_process so that my server will perform multiple child processes simultaneously?

message queue and back-end process.

i do exactly what you're wanting, using RabbitMQ. there are several other great messaging systems out there, like ZeroMQ and even Redis w/ some pub-sub libraries on top of it.

the gist of it is to send a request to your queueing system and have another process pick up the message, then run the process to do the work.

if you need a response from the worker, you can use bi-directional messaging with either a Request/Reply setup, or use status messages for really-long-running things.

if you're interested in the RabbitMQ side of things, I have a free email course on various patterns with RabbitMQ, including Request/Reply and status emails: http://derickbailey.com/email-courses/rabbitmq-patterns-for-applications/

and if you're interested in ground-up training on RMQ w/ Node, check out my training course at http://rabbitmq4devs.com

Upvotes: 2

Related Questions