CODEWITHSUNDEEP
perlfork
cppcoder
cppcoder

Reputation: 23095

Is there a limit in the number of processes that can be forked in perl?

I am facing a weird problem. The forked processes is not increasing more than 64.

sub create_process()
{
    my $child_pid;
    my @nsitr;

    my $i = 0;

    foreach my $domain (@domains)
    {
        @nsitr = @nameservers;
        open my $dnsfh, '>', $outputfile or die "Unable to open $outputfile - $!";
        foreach my $ns (@nameservers)
        {
            print "Forking child $i\n";
            defined($child_pid = fork() ) or (die "Unable to fork a new process" && next);
            $i++;
            if($child_pid == 0)
            {
                &resolve_dns($dnsfh, $domain, $ns);
                exit;
            }

        }
        close $dnsfh;
    }
}

Output

...
...
Forking child 60
Forking child 61
Forking child 62
Forking child 63
Forking child 64
Forking child 64
Forking child 64
Forking child 64
Forking child 64
...
...

Upvotes: 2

Views: 1698

Answers (3)

Oleg V. Volkov
Oleg V. Volkov

Reputation: 22421

If you're on some kind of Unix system, you may want to use threads module instead of forks. Actually it is even heavier in some aspects and may require some extra housekeeping, but this could allow your to surpass forked process limit. There could be other limits that'd prevent you from creating big amount of threads though.

Upvotes: 0

Sinan Ünür
Sinan Ünür

Reputation: 118118

If you want to do parallel lookups, you can either use the demo script included with Net::DNS or check out AnyEvent::DNS.

The latter provides

This module offers both a number of DNS convenience functions as well as a fully asynchronous and high-performance pure-perl stub resolver.

I haven't used it, but IO::Lambda::DNS would also allow parallel queries to be made:

# parallel async queries
   lambda {
      for my $site ( map { "www.$_.com" } qw(google yahoo perl)) { 
         context $site, 'MX', timeout => 0.25; 
         dns { print shift-> string if ref($_[0]) }
      }
   }-> wait;

Using those modules might be preferable to managing forks by hand.

Based on your comment, I think you might have misunderstood what I was trying to say. Maybe this will help:

#!/usr/bin/env perl

use strict; use warnings;

use AnyEvent::DNS;
use AnyEvent::Socket;
use YAML;

my %nameservers = (
    'Google' => '8.8.4.4',
    'Dnsadvantage' => '156.154.71.1',
    'OpenDNS' => '208.67.222.222',
    'Norton' => '198.153.194.1',
    'Verizon' => '4.2.2.4',
    'ScrubIt' => '207.225.209.66',
);

for my $ip ( values %nameservers ) {
    $ip = AnyEvent::DNS->new(
        server => [ parse_address($_) ],
        timeout => [3],
    );
}

my @domains = qw(example.com cnn.com bing.com);

my $cv = AnyEvent->condvar;
for my $domain (@domains) {
    for my $ns (keys %nameservers) {
        $cv->begin;
        $nameservers{$ns}->resolve(
            $domain, 'a', sub {
                $cv->end;
                print Dump { $ns => [ @{$_[0]}[0,4] ] };
            }
        );
    }
}

$cv->recv;

Output:

---
ScrubIt:
  - example.com
  - 192.0.43.10
---
ScrubIt:
  - cnn.com
  - 157.166.226.26
---
Norton:
  - example.com
  - 192.0.43.10
---
OpenDNS:
  - example.com
  - 192.0.43.10
---
Dnsadvantage:
  - example.com
  - 192.0.43.10
---
Verizon:
  - example.com
  - 192.0.43.10
---
Google:
  - example.com
  - 192.0.43.10
---
ScrubIt:
  - bing.com
  - 65.52.107.149
---
Norton:
  - cnn.com
  - 157.166.255.18
---
OpenDNS:
  - cnn.com
  - 157.166.255.19
---
Dnsadvantage:
  - cnn.com
  - 157.166.226.25
---
Verizon:
  - cnn.com
  - 157.166.226.26
---
Google:
  - cnn.com
  - 157.166.255.18
---
Norton:
  - bing.com
  - 65.52.107.149
---
OpenDNS:
  - bing.com
  - 65.52.107.149
---
Dnsadvantage:
  - bing.com
  - 65.52.107.149
---
Verizon:
  - bing.com
  - 65.52.107.149
---
Google:
  - bing.com
  - 65.52.107.149

Upvotes: 2

geekosaur
geekosaur

Reputation: 61369

Perl doesn't define such a limit, but most operating systems do. Use waitpid to reap children, or on Unix-like systems you can use sigaction (from the POSIX module) to ignore SIGCHLD with the SA_NOCLDWAIT flag to have the system reap children automatically. (Linux happens to let you omit SA_NOCLDWAIT, but you should use it anyway.)

Upvotes: 5

Related Questions