CODEWITHSUNDEEP
arraysmultithreadingperl
Majoris
Majoris

Reputation: 3191

Perl - threads join and system running out of memory

I posted couple of question for array and threads in my Perl code. Here is the code again. Based on the suggestions, I made changes and it seemed to be working. Still the code is running into a new issue. The machine is running out of memory. Many threads are finished and unjoined.

Code - 

#!/usr/bin/perl -w -I /opt/hypertable/0.9.7.3/lib/perl -I /opt/hypertable/0.9.7.3/lib/perl/gen-perl
use strict;
use IO::Socket;
use Geo::IP;
use threads qw(stringify);
use Net::NBName;
use Data::Dumper;
use Hypertable::ThriftClient;

# Syslog Variables and Constants
my $MAXLEN = 1524;
my $limit = 5; #for testing
my $sock;
my $thr;
# Start Listening on UDP port 514
$sock = IO::Socket::INET->new(LocalPort => '514', Proto => 'udp') || die("Socket: $@");

my $buf = '';
do{
my $count = 0;
my @set;

for ($count = 0; $count <= $limit; $count++) {
  $sock->recv($buf, $MAXLEN);
  my ($port, $ipaddr) = sockaddr_in($sock->peername);
  my $hn = gethostbyaddr($ipaddr, AF_INET);
  $buf =~ /<(\d+)>(.*?):(.*)/;
  my $msg = $3;
  $set[$count][0] = $hn;
  $set[$count][3] = $msg;
  print $count." --> ".$set[$count][0]." --> ".$set[$count][4]."\n";#Print original array, should print 5 elements 

  $thr = threads->create('logsys',@set);

  #&logsys(@set);
}
}while(1);

$thr->join();

sub logsys {
  my $count = 0;
  my @set = @_;
  my $geoip = Geo::IP->new(GEOIP_CHECK_CACHE);
  my $nb = Net::NBName->new;

  print "--------------------- ".scalar (@set)." -------------------\n";

  for ($count=0; $count <= $limit; $count++) {
    print $count." --> ".$set[$count][0]." --> ".$set[$count][5]."\n";#print passed array, should same exact 5 elements
    if (open(WW,">syslog")){
      print WW $count." --> ".$set[$count][0]." --> ".$set[$count][6]."\n"; close(WW);
    }
  }
}

Error - 

Out of memory!
Callback called exit at /usr/lib/perl/5.10/IO/Socket.pm line 287.
Thread 646 terminated abnormally: main=HASH(0xee3b3068) at ./syslogd.pl line 50.
Perl exited with active threads:
        9 running and unjoined
        644 finished and unjoined
        0 running and detached

What does this mean? Why the machine is running out of memory even when many threads are finished? How do I fix this? I can provide more info if needed.

Upvotes: 0

Views: 1496

Answers (1)

didierc
didierc

Reputation: 14730

You need to join or detach the threads to free up the memory they are retaining.

Just add :

$thr->detach();

after the thread creation, or make a dedicated thread to join the other threads once they are finished, if you need to retreive their result somehow.

Seeing your update of the code, you seem to want to join the created threads, but you try to do so outside the do loop, which does not seem to end. The join call is therefore never reached apparently. Even if it did, the $thr variable would only contain a reference to the latest created thread, and hence would only allow to join that very instance.

Given what I see of your code, I think the best course of action is really to have your threads run in detached state, and have them push to a queue whatever result they need to return.

Upvotes: 1

Related Questions