CODEWITHSUNDEEP
perlfork
Jeremy
Jeremy

Reputation: 5433

Perl - responsible forking

I have newly discovered Perl forking and I am quite in love. But one thing concerns me -- if I am just splitting off processes left-and-right, surely this will cause some issue somewhere. Is there a reasonable kind of check that one should use to make that sure that my little application doesn't eat up all of the resources of my machine?

Take this sample code:

foreach my $command (@commands) {
   my $pid = fork();
   if (!$defined $pid) {
     #Fork failed. Do something.
   } elsif ($pid == 0) { #This is the child.
      system($command);
      exit(0)
   }
}

while (wait() != -1) {} #wait() will be -1 when the last child exits.

So this will work fine, and spawn off a process to handle each command. It will all happen in parallel which is great if these commands are completely independent.

What if I suddenly have 5,000+ commands to run through? It wouldn't be wise to thoughtlessly fork off that many processes. So what kind of check should be implemented, and how?

)

Upvotes: 6

Views: 2202

Answers (4)

Brian Roach
Brian Roach

Reputation: 76918

When a child exits it's going to send back a SIG_CHLD to the parent. You'll want to reap these as if you don't they will be zombies in the process table until that final call to wait at the end of your script.

Chapter 16 of O'Reilly's Perl Cookbook on Google books provides a bunch of information on this. Basically, you need to increment a counter when you're forking children, and decrement it when you're reaping them, and not fork new ones past a reasonable max of currently running children.

As to what a "reasonable max" is ... depends on the hardware, and what those forked processes are doing. There's no static answer to that question other than to say test what you're doing and look at the performance impact on the machine. Preferably during business hours. After telling the sysadmin what you're doing. He/she may even have some advice.

Upvotes: 3

DVK
DVK

Reputation: 129569

Also if you are worried about spawning too many forked processes at the same time, you can throttle them.

Either roll your own (using a queue to hold "to fork" ones), or better yet, use Parallel::ForkManager module which allows you to limit simultaneous forks via a constructor parameter.

use Parallel::ForkManager;
$pm = new Parallel::ForkManager($MAX_PROCESSES);

Please note that ForkManager will ALSO take care of reaping ended child processes for you via a provided "wait*" APIs

Upvotes: 7

JRFerguson
JRFerguson

Reputation: 7526

Spawning (forking) as many processes as your system can support is called a fork bomb . Your system may freeze or crash as its process table becomes saturated and memory and CPU are depleted. On Linux systems, the ulimit command should show the maximum number of user processes permitted. You can (should) set this appropriately for your system. Forking creates processes exponentially. Thus, this snippet create four processes that endless consume CPU until killed --- a nasty, tiny bomb:

perl -e 'fork; fork; 1 while {}'

Upvotes: 0

Damodharan R
Damodharan R

Reputation: 1507

To make sure you dont spawn more processes than what system can efficiently handle you can use modules like Parallel::ForkManager

Upvotes: 1

Related Questions