CODEWITHSUNDEEP
pythonsubprocesspipe
Aran-Fey
Aran-Fey

Reputation: 43146

Why can a subprocess still write to stdout after it's been closed?

I found this piece of code in the subprocess documentation, where one process's stdout is being piped into another process:

p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

And I'm confused by that stdout.close() call. Surely closing the stdout handle prevents the process from producing any output?

So I ran an experiment, and to my surprise the process wasn't affected by it at all:

from subprocess import Popen, PIPE

p1 = Popen(['python', '-c', 'import time; time.sleep(5); print(1)'], stdout=PIPE)
p2 = Popen(['python', '-c', 'print(input()*3)'], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
print('stdout closed')
print('stdout:', p2.communicate()[0])

# output:
# stdout closed
# [5 second pause]
# stdout: b'111\n'

What's going on here? Why can the process write to a closed pipe?

Upvotes: 7

Views: 1166

Answers (2)

Davis Herring
Davis Herring

Reputation: 39808

Each process has its own set of file descriptors. You can (and should) close the parent’s copy once you don’t need it; the actual pipe (endpoint) survives until the last process (presumably p2) closes it. Only then does a process trying to write to the (other end of the) pipe get SIGPIPE.

If you don’t close it, the possibility remains that the parent will read from it even after p2 exits, which will needlessly keep p1 alive (and can cause deadlock if the parent waits on it).

Upvotes: 2

Kaz
Kaz

Reputation: 58568

Closing a file decriptor just means decrementing a reference count (inside the operating system kernel). The descriptor number becomes invalid, but nothing happens to the object it refers to unless the reference count hits zero.

Inside the Popen calls, operations are taking place which duplicate the file descriptor, thereby increasing the reference count: operations like dup/dup2 and fork.

If we fork a child process which receives a file descriptor from the parent, that file descriptor is a duplicate which points to the same object.

If the parent closes its original copy of that descriptor, that doesn't affect the one in the child. And vice versa. Only if both the parent and child close that descriptor does the underlying open file/device object go away; and only if there are no additional descriptors referencing it.

This is true even though the descriptors have the same number; each process has its own table of file descriptor numbers. File descriptor 3 in a child is different from the 3 in the parent.

Upvotes: 4

Related Questions