CODEWITHSUNDEEP
python-2.7subprocess
sandy_1111
sandy_1111

Reputation: 383

python resource limits

Is there some limit for usage of subprocess.Popen..? I observed that it fails continuously at 1017`th execution of external command.

usage: subprocess.Popen (cmd, shell=True, stdout=file_hndl, stderr=file_hndl)

Expecting the error and output to be redirected to file with file object file_hndl

Upvotes: 0

Views: 2113

Answers (2)

sandy_1111
sandy_1111

Reputation: 383

There is no fault with subprocess.Popen, the havoc is created by using the file_hndl in place of stdout and stderr.

All the resources are limited to each user process in operating system.

Ex:On Linux $ ulimit -a
 core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30254
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024

Note the number of files that can be opened is 1024, This was limiting the execution of subprocess.Popen.

set the resource limit as required using resource.setrlimit.

Ex: resource.setrlimit (resource.RLIMIT_NOFILE, (20000,20000))

Upvotes: 1

Alex Punnen
Alex Punnen

Reputation: 6234

You are running out of file handles. You can increase it and check; More info here

https://unix.stackexchange.com/questions/36841/why-is-number-of-open-files-limited-in-linux https://unix.stackexchange.com/questions/84227/limits-on-the-number-of-file-descriptors

Upvotes: 0

Related Questions