perl fork() doesn't seem to utilize the cores, but cpu only

Question

I've got a perl program that trying to do conversion of a bunch of files from one format to another (via a command-line tool). It works fine, but too slow as it's converting the files one and a time.

I researched and utilize the fork() mechanism trying to spawn off all the conversion as child-forks hoping to utilize the cpu/cores.

Coding is done and tested, it does improve performance, but not to the way I expected. When looking at /proc/cpuinfo, I have this:

> egrep -e "core id" -e ^physical /proc/cpuinfo|xargs -l2 echo|sort -u
physical id : 0 core id : 0
physical id : 0 core id : 1
physical id : 0 core id : 2
physical id : 0 core id : 3
physical id : 1 core id : 0
physical id : 1 core id : 1
physical id : 1 core id : 2
physical id : 1 core id : 3

That means I have 2 CPU and quad-core each? If so, I should able to fork out 8 forks and supposingly I should able to make a 8-min job (1min per file, 8 files) to finish in 1-min (8 forks, 1 file per fork).

However, when I test run this, it still take 4-min to finish. It appears like it only utilized 2 CPUs, but not the cores?

Hence, my question is:

1. Is it true that perl's fork() only parallel it based on CPUs, but not cores? Or maybe I didn't do it right? I'm simply using fork() and wait(). Nothing special.

2. I'd assume perl's fork() should be using cores, is there a simple bash/perl that I can write to prove my OS (i.e. RedHat 4) nor Perl is the culprit for such symptom?

To Add:

I even tried running the following command multiple times to simulate multiple processing and monitor htop.

while true; do echo abc >>devnull; done &

Somehow htop is telling me I've got 16 cores? and then when I spawn 4 of the above while-loop, I see 4 of them utilizing ~100% cpu each. When I spawn more, all of them start reducing the cpu utilization percentage evenly. (e.g. 8 processing, see 8 bash in htop, but using ~50% each) Does this mean something?

Thanks ahead. I tried google around but not able to find an obvious answer.

---

Edit: 2016-11-09

Here is the extract of perl code. I'm interested to see what I did wrong here.

my $maxForks = 50;
my $forks = 0;
while(<CIFLIST>) {
    extractPDFByCIF($cifNumFromIndex, $acctTypeFromIndex, $startDate, $endDate);
}
for (1 .. $forks) {
    my $pid = wait();
    print "Child fork exited.  PID=$pid\n";
}

sub extractPDFByCIF {
    # doing SQL constructing to for the $stmt to do a DB query
    $stmt->execute();

    while ($stmt->fetch()) {
        # fork the copy/afp2web process into child process
        if ($forks >= $maxForks) {
            my $pid = wait();
            print "PARENTFORK: Child fork exited.  PID=$pid\n";
            $forks--;
        }
        my $pid = fork;
        if (not defined $pid) {
            warn "PARENTFORK: Could not fork.  Do it sequentially with parent thread\n";
        }
        if ($pid) {
            $forks++;
            print "PARENTFORK: Spawned child fork number $forks. PID=$pid\n";
        }else {
            print "CHILDFORK: Processing child fork. PID=$$\n";
            # prevent child fork to destroy dbh from parent thread
            $dbh->{InactiveDestroy} = 1;
            undef $dbh;

            # perform the conversion as usual
            if($fileName =~ m/.afp/){
                    system("file-conversion -parameter-list");
            } elsif($fileName =~ m/.pdf/) {
                    system("cp $from-file $to-file");
            } else {
                    print ERRORLOG "Problem happened here\r\n";
            }
            exit;
        }
        # end forking

    $stmt->finish();
    close(INDEX);
}

Answer 1

fork() spawns a new process - identical to, and with the same state as the existing one. No more, no less. The kernel schedules it and runs it wherever.

If you do not get the results you're expecting, I would suggest that a far more likely limiting factor is that you are reading files from your disk subsystem - disks are slow, and contending for IO isn't actually making them any faster - if anything the opposite, because it forces additional drive seeks and less easy caching.

So specifically:

1/ No, fork() does nothing more than clone your process.

2/ Largely meaningless unless you want to rewrite most of your algorithm as a shell script. There's no real reason to think that it'll be any different though.

To follow on from your edit:

• system('file-conversion') looks an awful lot like an IO based process, which will be limited by your disk IO. As will your cp.

• Have you considered Parallel::ForkManager which greatly simplifies the forking bit?

• As a lesser style point, you should probably use 3 arg 'open'.

  #!/usr/bin/env perl
  use strict;
  use warnings;
  use Parallel::ForkManager;
  
  my $maxForks = 50;
  
  my $manager = Parallel::ForkManager->new($maxForks);
  
  while ($ciflist) {
  
      ## do something with $_ to parse.
  
      ##instead of: extractPDFByCIF($cifNumFromIndex, $acctTypeFromIndex, $startDate, $endDate);
  
      # doing SQL constructing to for the $stmt to do a DB query
      $stmt->execute();
  
      while ( $stmt->fetch() ) {
  
          # fork the copy/afp2web process into child process
          $manager->start and next;
          print "CHILDFORK: Processing child fork. PID=$$\n";
  
          # prevent child fork to destroy dbh from parent thread
          $dbh->{InactiveDestroy} = 1;
          undef $dbh;
  
          # perform the conversion as usual
          if ( $fileName =~ m/.afp/ ) {
              system("file-conversion -parameter-list");
          } elsif ( $fileName =~ m/.pdf/ ) {
              system("cp $from-file $to-file");
          } else {
              print ERRORLOG "Problem happened here\r\n";
          }
  
          # end forking
          $manager->finish;
      }
      $stmt->finish();
  
  }
  
  $manager->wait_all_children;
  

Answer 2

Your goal is to parallelize your application in a way that is using multiple cores as independent resources. What you want to achieve is multi-threading, in particular Perl's ithreads that are using calls to the fork() function of the underlying system (and are heavy-weight for that reason). You can teach the Perl way of multi-threading to yourself from perlthrtut. Quote from perlthrtut:

When a new Perl thread is created, all the data associated with the current thread is copied to the new thread, and is subsequently private to that new thread! This is similar in feel to what happens when a Unix process forks, except that in this case, the data is just copied to a different part of memory within the same process rather than a real fork taking place.

That being said, regarding your questions:

1. You're not doing it right (sorry). [see my comment...] With multi-threading you don't need to call fork() by yourself for that, but Perl will do it for you.

2. You can check whether your Perl interpreter has been built with thread support e.g. by perl -V (note the capital V) and looking at the messages. If there is nothing to see about threads then your Perl interpreter is not capable of Perl-multithreading.

The reason that your application is already faster even with only one CPU core by using fork() is likely that while one process has to wait for slow resources such as the file system another process can use the same core as a computation resource in the meantime.