CODEWITHSUNDEEP
pythonsubprocessstdoutstderr
Elmo
Elmo

Reputation: 6471

Get STDOUT, STDERR without waiting for process to exit

Some processes exit after a long time and their status is constantly being written to STDOUT.

How can I run a process and read its STDOUT without waiting for the process to exit?

I tried:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
o = p.communicate()[0] # Waits until process exits, also tried p.stdout.read()
print o                # Isn't printed until process exists
do_something(o)        # Doesn't get executed until process exits

Upvotes: 4

Views: 2349

Answers (1)

tdelaney
tdelaney

Reputation: 77407

Reading line by line is straight forward

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
for line in p.stdout:
    do work here
p.wait()
if p.returncode != 0:
    panic here

...mostly. Commands tend to buffer differently depending on whether they are run from a conole or another program. Since you are not a console, you may find that it outputs less often... but it'll all eventually get there before the program exits. On linux you can use pty or pexpect or something, but I know of no good solution on windows.

Now if you want to run them in parallel, create a thread per command you run and thread.join() each of them at the end of your programming. This is a bit tricky in itself. You could use a thread pool (see multiprocessing.ThreadPool on python 2.x, don't remember the name off hand in 3.x).

Upvotes: 2

Related Questions