CODEWITHSUNDEEP
perlparallel-processing
ohad kim
ohad kim

Reputation: 45

Parallel::ForkManager takes too much time to start 'finish' function

I use the regular Parallel::ForkManager module in Perl. I execute around 10 children processes. 'passes_thresholds' function takes few milliseconds or nanoseconds (checked it). In case I run all processes one by one (without Parallel::ForkManager) the whole process took 80-250 milliseconds. In case I run them as parallel, the whole process takes at least 1 second. I found that the Fork spends 1 second to start the 'finish' function. I put a timer when the child process finished his job and should go to 'finish'- function. One second is too much for my development.

sub parallel_execute {
    my $this = shift;
    foreach my $a (@a_array) {
        my $pid = $this->{fork_manager}->start and next;
        my $res = $a->passes_thresholds();
        $a->{timer} = Benchmark::Timer->new();
        $svc->{timer}->start;
        $this->{fork_manager}->finish(0,{a => $a, plugin_result => $res});
    }
}

$this->{fork_manager}->run_on_finish( sub {
    my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data_structure_reference) = @_;
    my $a = $data_structure_reference->{a};
    if (exists $a->{timer}) {
        $a->{timer}->stop;
        debug "took: " . $a->{timer}->report;
    }
});

Do you have any idea why it took at least 1 second to start the 'finish' command?

(I am using Unix server, and perl 5.10)

Upvotes: 0

Views: 562

Answers (2)

ohad kim
ohad kim

Reputation: 45

Thanks you all, i found the problem. ForManager module has 'waitpid_blocking_sleep' parameter, which defined as 1 second by default. There is a function called 'set_waitpid_blocking_sleep' which we can defined this parameter (sleep time). We can set zero or fraction of seconds. I set this paramter to zero and it fixed my issue.

Upvotes: 2

ikegami
ikegami

Reputation: 385565

The on_finish callback is only called when P::FM reaps a child, and P::FM only reaps a child under three conditions:

  • When $pm->start is called and the number of children that have been started but not reaped is equal to the maximum.
  • When $pm->reap_finished_children is called.
  • When $pm->wait_all_children is called.

There could be an arbitrarily long delay between a child exiting and one the above events. Adding the following to your program should eliminate that delay:

$SIG{CHLD} = sub { $pm->reap_finished_children };

By the way, if the work performed by your child only takes "a few milliseconds or nanoseconds", you are actually slowing things down by using P::FM. Data passed to finish gets serialized and written to disk, then read from the disk and deserialized for the on_finish callback!

Upvotes: 1

Related Questions