CODEWITHSUNDEEP
raku
con
con

Reputation: 6093

Perl6: Cannot invoke this object (REPR: P6opaque; Parallel::ForkManager)

I'm attempting to run a series of shell commands in parallel in Perl6, using Perl5's Parallel::ForkManager This is an almost exact translation of working Perl5 code.

CONTROL {
    when CX::Warn {
        note $_;
        exit 1;
    }
}
use fatal;
role KeyRequired {
    method AT-KEY (\key) {
        die "Key {key} not found" unless self.EXISTS-KEY(key);
        nextsame;
    }
}

use Parallel::ForkManager:from<Perl5>;

sub run_parallel (@cmd) {
    my $manager = Parallel::ForkManager(8).new();
    for (@cmd) -> $command  {
        $manager.start and $manager.next;
        my $proc = shell $command, :out, :err;
        if $proc.exitcode != 0 {
            put "$command failed";
            put $proc.out.slurp;
            put $proc.err.slurp;
            die;
        }
        $manager.finish;
    }
    $manager.wait_all_children;#necessary after all lists
}

my @cmd;
my Str $dir = 'A/1';
for dir($dir, test => /\.vcf\.gz$/) -> $vcf {
    @cmd.append: "aws s3 cp $vcf s3://s3dir/$dir/"
}
put @cmd.elems;
run_parallel(@cmd);

Basically, I'm trying to parallelize tedious shell commands.

However, this mysterious error comes up:

Cannot invoke this object (REPR: P6opaque; Parallel::ForkManager) in sub run_parallel at 2.aws_cp.p6 line 18 in block at 2.aws_cp.p6 line 39

Why is Perl6 saying this? what is wrong? how can I get these commands to run?

Perhaps there is a more native/idiomatic way to run shell commands in parallel in Perl6?

Upvotes: 3

Views: 83

Answers (2)

Scimon Proctor
Scimon Proctor

Reputation: 4558

You probably want to look at using Proc::Async which runs external commands asynchronously in threads without forking separate instances of the code to do it.

Upvotes: 7

Brad Gilbert
Brad Gilbert

Reputation: 34120

Perl5's Parallel::ForkManager probably won't work in Perl6 because of how Inline::Perl5 is implemented.

Inline::Perl5 embeds the Perl5 compiler/runtime inside of Perl6.

Parallel::ForkManager expects that Perl5 was run by itself.

If you ever did get it to do something other than generate an error it would probably screw up the Perl6 runtime. The main problem is the use of fork. For more information about why fork is a a problem see the article Bart Wiegmans (brrt) wrote about it: “A future for fork(2)”

Perl6 already has a similar feature that is easier to use.

sub run_parallel (@cmd) {
    my @children = do for (@cmd) -> $command  {
        start {
            my $proc = shell $command, :out, :err;
            if $proc.exitcode != 0 {
                put "$command failed";
                put $proc.out.slurp;
                put $proc.err.slurp;
                die;
            }
        }
    }
    await @children;
}

start is a prefix that tells the runtime to start running the following code sometime in the near future. It returns a Promise.
await takes a list of Promises and returns a list of their results.

start basically calls Promise.start which is similar to:

sub start ( &code ) {
    my $promise = Promise.new;
    my $vow = $promise.vow;

    $*SCHEDULER.cue(
        { $vow.keep(code(|c)) },
        :catch(-> $ex { $vow.break($ex); }) );
    $promise
}

So it will use the globally available thread pool in $*SCHEDULER. If you want to use a separate one you could.

sub run_parallel (@cmd) {
    my $*SCHEDULER = ThreadPoolScheduler.new(max_threads => 8);

    my @children = do for (@cmd) -> $command  {
        start {
            my $proc = shell $command, :out, :err;
            if $proc.exitcode != 0 {
                put "$command failed";
                put $proc.out.slurp;
                put $proc.err.slurp;
                die;
            }
        }
    }
    await @children;
}

It would make more sense to use Proc::Async for this though.

Upvotes: 5

Related Questions