CODEWITHSUNDEEP
pythonsubprocess
Caleb Spare
Caleb Spare

Reputation: 507

How can I call a command using Python, capturing stderr and stdout, without waiting for stderr/stdout to be closed?

Right now I have some code that uses Popen.communicate() from subprocess (setting stdin=PIPE and stderr=PIPE) to run a command and capture both stderr and stdout.

The problem is that communicate() not only waits for the command to exit, it waits for stdout and stderr to be closed. The command I'm running spawns a child process which keeps stderr open, so even though the command is finished running (and shows as "defunct" in ps) communicate() is still hung.

I want to only wait for the command to finish without waiting on stderr/stdout. But I still want to capture any stderr/stdout output given while the command was running. The documentation for wait() is accompanied by a red box with a disclaimer:

This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

Obviously, I also want to avoid deadlocks.

What is the right way to accomplish this task?

Upvotes: 5

Views: 1150

Answers (2)

jfs
jfs

Reputation: 414865

"shows as "defunct" in ps" implies that you might be on a system where select or fcntl would work i.e., you can read stdout/stderr without blocking easily.

Example: A starts B (cmd, the child), B starts C (grandchild), A reads output until B exits or EOF:

#!/usr/bin/env python
import os
from select import select
from subprocess import Popen, PIPE

p = Popen(cmd, stdout=PIPE, stderr=PIPE, bufsize=0)
read_set = [p.stdout, p.stderr]
pipename = {p.stdout: "stdout", p.stderr: "stderr"}
timeout = 0.5 # ugly but it works
while read_set and p.poll() is None: # while subprocess is running or until EOF
    for pipe in select(read_set, [], [], timeout)[0]:
        data = os.read(pipe.fileno(), 1<<30)
        if data:
            print("got from %s: %r" % (pipename[pipe], data))
        else: # EOF
            pipe.close()
            read_set.remove(pipe)
print("exit code %s" % (p.wait(),))

# child exited, wait for grandchild to print
for pipe in read_set:
    print("read the rest of %s: %r" % (pipename[pipe], pipe.read()))
    pipe.close()

where cmd:

import sys
from textwrap import dedent

cmd = [sys.executable, '-u', '-c', dedent("""
    # inception style
    import os
    import sys
    from subprocess import Popen
    from textwrap import dedent

    Popen([sys.executable, '-u', '-c', dedent('''
        import os
        import sys
        import time

        time.sleep(60)
        print("grandchild %d done" % os.getpid())
        sys.stderr.write("grandchild stderr")
        sys.exit(20)
    ''')]) # stdout/stderr are not redirected

    print('child %d done' % os.getpid())
    sys.stderr.write('child stderr')
    sys.exit(19)
""")]

Upvotes: 1

Nodari L
Nodari L

Reputation: 1178

I've tried a simple example. And all I've got so far is capturing of the std.err I've created two files. The contents of the first is (example1.py):

import time
import sys

for i in xrange(1000):
    print >>sys.stderr, "hello", i
    time.sleep(1)

The contents of the second one(example2.py):

import subprocess

cmd = "python example1.py"

print subprocess.check_output(cmd, shell=True)

And that I've just called the example2.py script:

python example2.py And I've realtime output ( I assume it is true:) ):

hello 0
hello 1
hello 2
hello 3
hello 4
hello 5
hello 6

Still, I have no idea, how to deal with standard output, but if I manage it, I will post the answer here

Upvotes: 0

Related Questions