Andy
Andy

Reputation: 780

Using maximum threads on multicore cpu

at first, my perlskills are limited so please keep that in mind. ;)

I have written a perlscript which indexes a directory and does something for each file in it.

Processing a file takes some time. Between 1 - 5 mins each at a cpu load nearly 100% (one core). My idea is, cause i have a quadcore cpu, to process more then one file at once, which leads me to perl threads.

So here are my questions.

1) Is my assumption right, that perl threads are allocated automatically to multiple cores?

2) I found this code example which does what i need, i think but i cannot figure out, how to keep always only 8 threads active. The example starts a static count of threads and is done when they are processed. However, in my case i have, lets say, 50 files to process but only 8 threads should be active at the same time.

So it should be like this: Read the directory, start 8 threads for the 8 first files and keep 8 threads working until all files are processed.

#!/usr/local/roadm/bin/perl
# This is compiled with threading support

use strict;
use warnings;
use threads;
use threads::shared;

print "Starting main program\n";

my @threads;
for ( my $count = 1; $count <= 10; $count++) {
        my $t = threads->new(\&sub1, $count);
        push(@threads,$t);
}   
foreach (@threads) {
        my $num = $_->join;
        print "done with $num\n";
}   
print "End of main program\n";

sub sub1 {
        my $num = shift;
        print "started thread $num\n";
        sleep $num;
        print "done with thread $num\n";
        return $num;
}   

Source: https://wiki.bc.net/atl-conf/pages/viewpage.action?pageId=20548191

I searched some hours now but did not find an example, how to do this. Would be nice when someone could give me a hint how to start.

Thank you.

Upvotes: 3

Views: 4749

Answers (4)

user12574431
user12574431

Reputation: 1

#!/usr/bin/env perl -w
use strict;
use threads;

my $cpu ||= 5;
my @threads;
my @all_target = qw /1 2 3 4 5 6 7 8 9/;

while(my($t,$job) = each @all_target){

    my $thread = threads -> create(\&do,$job);
    push @threads,$thread;

    if(@threads == $cpu or $t == $#all_target){
        for(@threads){
            $_ -> join();
            print "$_ join\n";
        }
        @threads = ();
    }
}

sub do{
    my $be_do = shift;

    print"$be_do\n";
    sleep 2;
#   $cmd = "do what";
#   system("$cmd");
}

Upvotes: 0

user12574431
user12574431

Reputation: 1

#!/usr/bin/env perl -w
use strict;
use threads;

my $cpu ||= 5;
my @threads;
my @all_target = qw /1 2 3 4 5 6 7 8 9/;

while(my($t,$job) = each @all_target){
    $t = threads -> create(\&do,$job);
    push @threads,$t;

    if(@threads == $cpu or $t == $#all_target){
        for(@threads){
            $_ -> join();
        }
        @threads = ();
    }
}`enter code here`

sub do{
    my $be_do = shift;
    print"$be_do\n";
    sleep 2;
}

Upvotes: 0

ysth
ysth

Reputation: 98398

Threads in perl are heavyweight; they take time and cpu to start and time, cpu, and memory to share data between them (and require that you carefully check what modules you are using for thread safety).

You are often much better off forking; Parallel::ForkManager makes this easy.

Upvotes: 6

bvr
bvr

Reputation: 9697

Good solution to your problem using Thread::Queue and number of workers can be found in this answer.

Upvotes: 5

Related Questions