CODEWITHSUNDEEP
pythonsubprocessblocking
Alfe
Alfe

Reputation: 59456

Gather subprocess output nonblocking in Python

Is there an easy way of gathering the output of a subprocess without actually waiting for it?

I can think of creating a subprocess.Popen() with capturing its stdout, then call p.communicate(), but that would block until the subprocess terminates.

I can think of using subprocess.check_output() or similar, but that also would block.

I need something which I can start, then do other stuff, then check the subprocess for being terminated, and in case it is, takes its output.

I can think of two rather complicated ways to achieve this:

  1. Redirect the output into a file, then after termination I can read the output from that file.
  2. Implement and start a handler thread(!) which constantly tries to read data from the stdout of the subprocess and adds it to a buffer.

The first one needs temporary files and disk I/O which I do not really like in my case. The second one means implementing quite a bit.

I guess there might be a simpler way I couldn't think of yet, or a ready-to-be-used solution in some library I didn't find yet.

Upvotes: 2

Views: 292

Answers (2)

Jean-François Fabre
Jean-François Fabre

Reputation: 140196

What's wrong with calling check_output in a thread?

import threading,subprocess

output = ""

def f():
    global output
    output = subprocess.check_output("ls")  # ["cmd","/c","dir"] for windows


t = threading.Thread(target=f)
t.start()
print('Started')
t.join()
print(output)

note that one could be tempted to use p = subprocess.Popen(cmd,stdout=subprocess.PIPE), wait for p.poll() to be != None and try to read p.stdout afterwards: that only works when the output is small, else you get a deadlock because stdout buffer is full and you have to read it from time to time.

Using p.stdout.readline() would work but would also block if the process doesn't print on a regular basis. If your application prints to the output all the time, then you can consider it as non-blocking and the solution is acceptable.

Upvotes: 1

Praind
Praind

Reputation: 1571

I think what you want is an unbuffered stdout stream. With that you will be able to capture the output of your process without waiting for it to finish. You can achieve that with the subprocess.Popen() function and the parameter stdout=subprocess.PIPE.

Try something like this

proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)

line = proc.stdout.readline()
while line:
    print line
    line = proc.stdout.readline()

Upvotes: 0

Related Questions