What difference between subprocess.call() and subprocess.Popen() makes PIPE less secure for the former?

Question

I've had a look at the documentation for both of them.

This question is prompted by J.F.'s comment here: Retrieving the output of subprocess.call()

The current Python documentation for subprocess.call() says the following about using PIPE for subprocess.call():

Note Do not use stdout=PIPE or stderr=PIPE with this function. The child process will block if it generates enough output to a pipe to fill up the OS pipe buffer as the pipes are not being read from.

Python 2.7 subprocess.call():

Note Do not use stdout=PIPE or stderr=PIPE with this function as that can deadlock based on the child process output volume. Use Popen with the communicate() method when you need pipes.

Python 2.6 includes no such warnings.

Also, the subprocess.call() and subprocess.check_call() don't seem to have a way to access their output, except for using stdout=PIPE with communicate():

https://docs.python.org/2.6/library/subprocess.html#convenience-functions

Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.

https://docs.python.org/2.6/library/subprocess.html#subprocess.Popen.communicate

What difference between subprocess.call() and subprocess.Popen() makes PIPE less secure for subprocess.call()?

More Specific: Why does subprocess.call() "deadlock based on the child process output volume.", and not Popen()?

Answer 1

Both call and Popen provide means of accessing the output of your command:

• With Popen you can use communicate or provide a file descriptor or file object to the stdout=... parameter.
• With call your only option is to pass a file descriptor or a file object to the stdout=... parameter (you can not use communicate with this one).

Now, the reason why stdout=PIPE is insecure when used with call is because call doesn't return until the subprocess has finished, this means all the output would have to reside in memory until that moment, and if the amount of output is too much then that would fill the OS pipes' buffer.

The references where you can validate the above information are the following:

1. According to this the parameters for both call and Popen are the same:

The arguments shown above are merely the most common ones, described below in Frequently Used Arguments (hence the slightly odd notation in the abbreviated signature). The full function signature is the same as that of the Popen constructor - this functions passes all supplied arguments directly through to that interface.

2. According to this the possible values for the stdout parameter are:

Valid values are PIPE, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created

Answer 2

call() is just Popen().wait() (± error handling).

You should not use stdout=PIPE with call() because it does not read from the pipe and therefore the child process will hang as soon as it fills the corresponding OS pipe buffer. Here's a picture that shows how data flows in command1 | command2 shell pipeline:

It does not matter what your Python version is -- the pipe buffer (look at the picture) is outside of your Python process. Python 3 does not use C stdio but it affects only the internal buffering. When the internal buffer is flushed the data goes into the pipe. If command2 (your parent Python program) does not read from the pipe then command1 (the child process e.g., started by call()) will hang as soon as the pipe buffer is full (pipe_size = fcntl(p.stdout, F_GETPIPE_SZ) ~65K on my Linux box (max value is /proc/sys/fs/pipe-max-size ~1M)).

You may use stdout=PIPE if you read from the pipe later e.g., using Popen.communicate() method. You could also read from process.stdout (the file object that represents the pipe) directly.