CODEWITHSUNDEEP
perlfilesolarisulimit
Arav
Arav

Reputation: 5247

Problems opening more than 10,000 files in Perl

I need to open more than 10,000 files in a Perl script, so I asked the system administrator to change the limit on my account to 14,000. ulimit -a now shows these settings:

core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 14000
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited

After the change I ran a test Perl program that opens/creates 256 files and closes 256 file handles at the end of script. When it creates 253 files the program dies saying too many open files. I don't understand why I'm getting this error.

I am working on a Solaris 10 platform. This is my code

my @list;
my $filename = "test";

for ($i = 256; $i >= 0; $i--) {
    print "$i " . "\n";
    $filename = "test" . "$i";
    if (open my $in, ">", ${filename}) {
        push @list, $in;
        print $in $filename . "\n";
    }
    else {
        warn "Could not open file '$filename'. $!";
        die;
    }
}

for ($i = 256; $i >= 0; $i--) {
    my $retVal = pop @list;
    print $retVal . "\n";
    close($retVal);
}

Upvotes: 7

Views: 2612

Answers (4)

Heto
Heto

Reputation: 595

you have a limit of 256 files. You forgot about STDIN, STDOUT and STDERR. Your 253 + default 3 = 256.

Upvotes: 0

David L.
David L.

Reputation: 2103

You might be able to work around the limitation with the FileCache Core module (keep more files open than the system permit).

Using cacheout instead of open, i was able to open 100334 files on linux:

#david@:~/Test$ ulimit -n
1024

#david@:~/Test$ perl plimit.pl | head
100333 
100332 
100331 
100330 
100329 

#david@:~/Test$ perl plimit.pl | tail
test100330
test100331
test100332
test100333

#david@:~/Test$ ls test* | wc -l
100334


modified version of your script (plimit.pl)

my @list;

use FileCache;

$mfile=100333;

my $filename="test";
for($i = $mfile; $i >= 0; $i--) {
    print "$i " . "\n" ;
    $filename = "test" . "$i";
    #if (open my $in, ">", ${filename}) {
     if ($in = cacheout( ">", ${filename}) ) {
        push @list,$in;
        print $in  $filename . "\n";
    } else {
        warn "Could not open file '$filename'. $!";
        die;
    }
}
for($i = $mfile; $i >= 0; $i--) {
    my $retVal = pop @list;
    print $retVal . "\n";
    close($retVal);
}

update

FileCache automatically closes and re-opens files if you exceed your system's maximum number of file descriptors, or the suggested maximum maxopen (NOFILE defined in sys/param.h).

In my case, on a linux box, it is 256:

#david@:~/Test$ grep -B 3 NOFILE /usr/include/sys/param.h 

/* The following are not really correct but it is a value 
   we used for a long time and which seems to be usable.  
   People should not use NOFILE and NCARGS anyway.  */
#define NOFILE      256

Using the lsof (list open files) command, the modified version of your script opened at most 260 of the 100334 files:

#david@:~/Test$ bash count_of_plimit.sh
20:41:27 18
new max is 18
20:41:28 196
new max is 196
20:41:29 260
new max is 260
20:41:30 218
20:41:31 258
20:41:32 248
20:41:33 193
max count was 260


count_of_plimit.sh

 #!/bin/bash
 # count open files with lsof
 #
 # latest revision: 
 #   ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/
 # latest FAQ: 
 #  ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ

 perl plimit.pl > out.txt &
 pid=$!

##adapted from http://stackoverflow.com/a/1661498
HOW_MANY=0
MAX=0
while [ -r "/proc/${pid}" ]; 
do
    HOW_MANY=`lsof -p ${pid} | wc -l`
    #output for live monitoring
    echo `date +%H:%M:%S` $HOW_MANY
    # look for max value
    if [ $MAX -lt $HOW_MANY ]; then
        let MAX=$HOW_MANY
        echo new max is $MAX
    fi 
    # test every second
    sleep 1
done
echo max count was $MAX

Upvotes: 8

ikegami
ikegami

Reputation: 386331

Tested with both your program and the following simpler program on a Windows box and a linux box without encountering the error you describe.

my @files;
for (;;) {
   print 1+@files, "\n";
   open my $fh, '<', $0 or die $!;
   push @files, $fh;
   last if @files == 500;
}

Output:

1
2
...
498
499
500

I don't think it's a Perl limitation, but a system limitation.

Note that it fails when you try to open the process's 257th handle (STDIN + STDOUT + STDERR + 253 = 256), which leads me to believe the number of open file handles a process can have must fit in 8 bits on your system. You could try verifying this by writing an equivalent C program and running it on the same machine.

#include <stdio.h>
#include <stdlib.h>

int main() {
   int i = 0;
   for (;;) {
      ++i;
      printf("%d\n", i);
      if (fopen("/bin/sh", "r") == NULL) {
         perror("fopen");
         exit(1);
      }

      if (i == 500)
         break;
   }

   return 0;
}

Upd: This has been confirmed here. Thanks, Schwern.

Upvotes: 4

Schwern
Schwern

Reputation: 165200

According to this article this is a default limitation of 32-bit Solaris. A program is normally limited to using the first 256 file numbers. STDIN, STDOUT and STDERR take 0, 1 and 2 which leaves you with 253. It's not a simple process to work around it, ulimit won't do it, and I don't know if Perl will honor it.

Here's a discussion about it on Perlmonks with a few suggested work arounds such as FileCache.

While the Solaris limitation is unforgivable, in general having hundreds of open filehandles indicates that your program could be designed better.

Upvotes: 16

Related Questions