CODEWITHSUNDEEP
node.jsspawnchild-process
bcarson
bcarson

Reputation: 21

Too many open files when using NodeJS child_processes.spawn to run scripts

Scenario:

Using a master script to spawn a variable number of child processes a variable number of times in order to perform load testing against a server.

The master script initially spawns all the children it can (according to its configuration settings) and then as the children processes exit if there are more runs requested by the config then new children are spun up.

What I'm seeing is an immediate failure upon attempting to spin up the 83rd child process. 83?

I'm not doing anything to explicitly close the files opened as part of the child spawning process but presumably that's not the job of the opening code but the child_processes module code?

I'm very curious about the magic number of 82 child processes. This seems to indicate something about either a limitation in node or some combination of node on my system?

Ideally, there's some lack of knowledge I have that this question will answer or someone can suggest an alternative way to launch child processes of scripts that won't suffer this issue?

I'm also interested in learning about the status of the Web Worker API that is coming to NodeJS. Anyone know anything about that?

The Details:

Thanks for any help.

Upvotes: 2

Views: 1494

Answers (1)

Peter Lyons
Peter Lyons

Reputation: 146014

My guess is the system is doing exactly what you are telling it. 82 processes is 3 open files per process. STDIN, STDOUT, STDERR. Bang. You've hit your ulimit just with the standard 3 file descriptors. Run with ulimit -n 512 and I bet you'll be able to run twice as many children.

Upvotes: 3

Related Questions