CODEWITHSUNDEEP
multithreadingperl
chris01
chris01

Reputation: 12431

Perl: share complex data-structure between threads

I like to share a complex-datastructure between threads. As far I know that is not possible with threads:shared (only basic types are shareable).

So I think about serialize/deserialize the structure with JSON or Storable so it is just a string that I can share perfectly. But I need to unpack it before use and pack it after a change.

Thanks for help!

EDIT

I just did some tests with Storable and JSON. JSON is quicker and produces smaller serialized strings. I did not expect that.

Upvotes: 4

Views: 415

Answers (2)

zdim
zdim

Reputation: 66964

Complex data structures can be shared using shared_clone. The components of the data structure need be cloned before being added to it.

use strict;
use feature 'say';
use Data::Dump qw(dd);

use threads;
use threads::shared;

my $cds = {
    k1 => shared_clone( { k1_l2 => [ 1..2 ] } ),
    k2 => shared_clone( { k2_l2 => [10..11] } )
};

my @threads = map { async(\&proc_ds, $cds->{$_}) } keys %$cds;

$_->join() for @threads;

dd $cds;
    
sub proc_ds {
    my ($ds) = @_;
    lock $ds;
    push @{$ds->{$_}}, 10+threads->tid  for keys %$ds;
}

Note that you don't want to allow autovivification when working with shared values, as it would create unshared (and empty) components in the structure. Check explicitly for existence.

A ready data structure needs to be cloned-and-shared

my $cds = { k => [ 5..7 ] };          # already built, need be shared
my $cds_share = shared_clone( $cds );

my @threads = map { async(\&proc_ds, $cds_share) } 1..3;
$_->join() for @threads;

With the same proc_ds() as above this prints the structure (condensed output)

{ 'k' => [ '5', '6', '7', '11', '12', '13' ] };

When data structure is populated for sharing, as in the first example, then there is less overhead to pay. Otherwise there is a data copy involved, as in the second example, and whether that is OK depends on details (data size, how often a copy is made, etc).

The idea of serializing data is workable as well, but how suitable it is again depends on details since in that case you'd not only copy data but would go to disks as well.

In that case JSON is certainly one good way to go, being a data format that is simple and readable, and can also be shared between tools. The Storable is binary, works directly with Perl data structures, and is supposed to be fast (what should show with larger data).

One other option is to use a worker model and pass data over a message queue. Then you'd use Thread::Queue, or perhaps make use of Thread::Queue::Any, for communication channels.

Upvotes: 2

Sobrique
Sobrique

Reputation: 53498

When dealing with this problem, I use Thread::Queue to pass my objects around, and usually use Storable to serialise.

I haven't bothered doing performance comparisons, because usually my data-passing overhead isn't the limiting factor.

Note - the key advantage of Storable is that it allows some limited object support (Not - be careful - it only works if your object is self contained):

#!/usr/bin/env perl
use strict;
use warnings;

package MyObject;

sub new { 
   my ( $class, $id ) = @_; 
   my $self = {};
   $self -> {id} = $id; 
   $self -> {access_count} = 0; 
   bless $self, $class;
   return $self;
}

sub access_thing { 
   my ( $self ) = @_;
   return $self -> {access_count}++; 
}

sub get_id { 
    my ( $self ) = @_;
   return $self -> {id}; 
}

package main; 

use threads;
use Thread::Queue;

use Storable qw ( freeze thaw );

my $thread_count = 10;

my $work_q = Thread::Queue -> new; 

sub worker  {
   while ( my $item = $work_q -> dequeue ) {
      my $obj = thaw ( $item ); 
      print $obj -> get_id, ": ", $obj -> access_thing,"\n";    

   }
}

for (1..$thread_count) {
   threads -> create (\&worker); 
}

for my $id ( 0..1000 ) {
   my $obj = MyObject -> new ( $id ); 
   $work_q -> enqueue ( freeze ( $obj ) );
}

$work_q -> end;

$_ -> join for threads -> list;

If JSON would limit you to array/hash data structures - which may be fine for your use case.

Upvotes: 3

Related Questions