CODEWITHSUNDEEP
multithreadingperlthread-safetythreadpool
tjwrona
tjwrona

Reputation: 9035

Can I reuse joined threads in perl?

I have a module that is running multiple threads and pushing them onto a list of threads.

ex:

#!/usr/bin/perl

#test_module.pm

package test_module;

use strict;
use warnings;
use threads;

sub main {

    my $max_threads = 10;
    my @threads     = ();

    # create threads
    while (scalar @threads < $max_threads) {
        my $thread = threads->new(\&thread_sub);
        push @threads, $thread;
    }

    # join threads
    for my $thread (@threads) {
        $thread->join();
    }
}

sub thread_sub {
    my $id = threads->tid();
    print "I am in thread $id\n";
}

1;

The problem is that I am calling this module multiple times from one Perl script and instead of eliminating the old threads and creating new ones, the thread ids just keep incrementing. I have heard that if you don't properly get rid of old threads in Perl this will cause a memory leak and slow your program down, is this true? Is the data from my old threads just sitting in memory taking up space?

If so this can become a large problem since my script will be part of a much larger program that may generate hundreds or thousands of threads all of which would just be taking up memory even after they are done being used. How can I stop this from happening? Can my threads be reused?

Here is an example script that will call the module and show how the threads will continue to increment even though I joined the old threads (I thought that "join" was how you cleaned up after them, am I doing something wrong?) The way this script will be used I can't afford to have memory from old threads sitting there taking up space.

ex:

#!/usr/bin/perl

#testing.pl

use strict;
use warnings;
use test_module;

test_module::main();
test_module::main();
test_module::main();

system 'pause';

Thanks!

Upvotes: 1

Views: 467

Answers (1)

Sobrique
Sobrique

Reputation: 53488

Don't worry about thread IDs incrementing - that doesn't mean the number of running threads is increasing. Once a thread is joined it has finished executing and been terminated.

However, continuously respawning threads isn't ideal either - creating a thread isn't a particularly lightweight operation in perl. So if you've got to do something like that, and are particularly focussing on efficiency - look to fork() instead.

I find I tend to use a 'worker thread' model, using Thread::Queue:

my $processing_q = Thread::Queue -> new();

sub worker_thread {
    while ( my $item = $processing_q -> dequeue() ) {
       # do stuff to $item
    }
}

for ( 1 .. $num_threads ) {
    my $thr = threads -> create ( \&worker_thread );
}

$processing_q -> enqueue ( @generic_list_of_things ); 
$processing_q -> end;

foreach my $thread ( threads -> list() ) {
    $thread -> join();
}

This will feed in a batch of items into a queue, and your worker threads will process them one at a time - means you can have a sensible number running, without having to continuously respawn.

As an alternative though - take a look at Parallel::ForkManager - fork style parallel processing may seem counterintuitive initially, but fork() is a native system call on Unix systems, so it tends to be better optimised.

Upvotes: 4

Related Questions