CODEWITHSUNDEEP
multithreadingperlhashmultidimensional-array
waltz777
waltz777

Reputation: 25

Sharing a 2 Dimensional Hash using Perl Threads

I'm using multi-threading to parse IHS log files. I'm assigning a separate thread for each file handle and counting the number of 500 errors.

sub parse_file { 

  my $file = shift; 
  my @srv = split /\//,$file;
  my $filename = $srv[$#srv];
  my $TD = threads->tid();

  $sem->down;
    print "Spawning thread $TD to process file \"$filename\"\n" if ($verbose);
    $rTHREADS++;
    $TIDs{$TD} = 1;
  $sem->up;

  open (FH, "$file") || die "Cannot open file $file $!\n";  
  while (<FH>){    
    if (/^(\d{13}).*?(\d{3}) [\-0-9] \d+ \d+ \//){   
      my $epoch = $1/1000; 
      my $http_code = $2;
      my $ti = scalar localtime($epoch);
      $ti =~ s/(\d{2}):\d{2}:\d{2}/$1/;

      if ($http_code eq '500'){
        unless ( exists $error_count{$ti} && exists $error_count{$ti}{$http_code} ){
          lock(%error_count);
          $error_count{$ti} = &share({});
          $error_count{$ti}{$http_code}++;
        }
      }
    }
  }
  close (FH);  

  $sem->down;
    print "Thread [$TD] exited...\n" if ($verbose);
    $rTHREADS--;
    delete $TIDs{$TD};
  $sem->up;

}

Problem is, the output looks like this using print Dumper(%http_count):

$VAR1 = 'Mon Apr 30 08 2012';
$VAR2 = {
          '500' => '1'
        };
$VAR3 = 'Mon Apr 30 06 2012';
$VAR4 = {
          '500' => '1'
        };
$VAR5 = 'Mon Apr 30 09 2012';
$VAR6 = {
          '500' => '1'
        };
$VAR7 = 'Mon Apr 30 11 2012';
$VAR8 = {
          '500' => '1'
        };
$VAR9 = 'Mon Apr 30 05 2012';
$VAR10 = {
           '500' => '1'
         };
$VAR11 = 'Mon Apr 30 07 2012';
$VAR12 = {
           '500' => '1'
         };
$VAR13 = 'Mon Apr 30 10 2012';
$VAR14 = {
           '500' => '1'
         };
$VAR15 = 'Mon Apr 30 12 2012';
$VAR16 = {
           '500' => '1'
         };

Job took 79 seconds

The 500 count for each date is always set to 1. I cannot get it to display the proper count. It seems that the statement $error_count{$ti} = &share({}); is the culprit but I'm not sure how to get around it.

Thanks!

Upvotes: 1

Views: 520

Answers (2)

Lumi
Lumi

Reputation: 15284

$error_count{$ti} = &share({});

You're assigning a new hash reference each time around, then incrementing the count in the following line. Change it to this:

$error_count{$ti} ||= &share({});

This will conditionally initialize the hashtable member. To be precise, it'll take effect when the value is either undef, 0 or the empty string.

Upvotes: 0

Greg Bacon
Greg Bacon

Reputation: 139551

According to the logic in your code, each value is incremented exactly once: when it does not yet exist in %error_count.

To increment the value every time through but create scaffolding only as necessary (which you have to do with shared containers rather than relying on autovivification), use

if ($http_code eq '500') {
  lock(%error_count);

  unless (exists $error_count{$ti} && exists $error_count{$ti}{$http_code}) {
    $error_count{$ti} = &share({});
  }

  $error_count{$ti}{$http_code}++;
}

If locking the entire hash is too broad a brush, look into using Thread::Semaphore instead.

Upvotes: 1

Related Questions