CODEWITHSUNDEEP
pythonsubprocess
Hugh
Hugh

Reputation: 761

subprocess.Popen : Separating stdout/stderr but keeping order

How do I get the output from a command using subprocess.Popen and have separate callbacks for stdout and stderr, but ensure that those callbacks are called in the order that the lines came from the process?

If I didn't care about separating out STDOUT and STDERR then I could do:

fd = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT )
line = fd.stdout.readline()
while line :
    callback( line )
    line = fd.stdout.readline()

However, if I have stdoutCallback and stderrCallback, and want them called on the appropriate outputs, but in the same order as the above code would call callback on, how would I go about doing this?

Upvotes: 5

Views: 2407

Answers (3)

jfs
jfs

Reputation: 414875

It is impossible. There is no order defined if writes are performed to different files.

You can get the correct order if the writes to stdout, stderr go to the same place (like in your stdout=PIPE, stderr=STDOUT case).

If "approximate" order is enough; here's a simple code example with threads and here's single-threaded version with a select-loop.

Upvotes: 2

Hugh
Hugh

Reputation: 761

So I think I've rolled my own with a couple of threads.

For the example below, test.py is this:

#!/usr/bin/python -u

import sys
import time

sys.stdout.write("stdout 1\n")
time.sleep(1)
sys.stderr.write("stderr 2\n")
time.sleep(1)
sys.stdout.write("stdout 3\n")
time.sleep(1)
sys.stderr.write("stderr 4\n")
time.sleep(1)

My code for getting the right output is:

#!/usr/bin/env python

import subprocess
from threading import Thread, Lock

cmdOutput = []
cmdOutputLock = Lock()
STDOUT = 1
STDERR = 2

def _outputLoop( fd, identifier ) :
    line = fd.readline()
    while line :
        cmdOutputLock.acquire()
        cmdOutput.append( ( line, identifier ) )
        cmdOutputLock.release()
        line = fd.readline()

p = subprocess.Popen( "test.py",
                      stdout = subprocess.PIPE,
                      stderr = subprocess.PIPE )

Thread( target=_outputLoop, args=( p.stdout, STDOUT ) ).start()
Thread( target=_outputLoop, args=( p.stderr, STDERR ) ).start()

while fd.poll() is None or cmdOutput :
    output = None
    cmdOutputLock.acquire()
    if cmdOutput :
        output = cmdOutput[0]
        del cmdOutput[0]
    cmdOutputLock.release()

    if output :
        if output[1] == STDOUT :
            print "STDOUT : {}".format( output[0].rstrip() )
        elif output[1] == STDERR :
            print "STDERR : {}".format( output[0].rstrip() )

I can definitely imagine a time when an stderr line is mixed up with an stdout line, but for the sake of what I wanted it for, it certainly works. (I was putting this as part of a logging module which would run a command and use different log levels for stdout and stderr.)

Upvotes: 1

vks
vks

Reputation: 67988

fd = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
output,error = fd.communicate()

Use communicate

Upvotes: 0

Related Questions