CODEWITHSUNDEEP
pythoniosubprocess
Jason S
Jason S

Reputation: 189646

Alternative to subprocess.Popen.communicate() for streaming

If I'm using subprocess.Popen I can use communicate() for small outputs.

But if the subprocess is going to take substantial time and produce substantial output, I want to access it as streaming data.

Is there a way to do this? The Python docs say

Warning: Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

I would really like to access a process output as a file-like object:

with someMagicFunction(['path/to/some/command','arg1','arg2','arg3']) as outpipe:
   # pass outpipe into some other function that takes a file-like object

but can't figure out how to do this.

Upvotes: 1

Views: 2029

Answers (1)

tdelaney
tdelaney

Reputation: 77337

communicate is a convenience method that starts background threads to read stdout and stderr. You can just read stdout yourself, but you need to figure out what to do with stderr. If you don't care about errors, you could add the param stderr=open(os.devnull, 'wb') or to a file stderr=open('somefile', 'wb'). Or, create your own background thread to do the read. It turns out that shutil already has such a function, so we can use it.

import subprocess
import threading
import shutil
import io

err_buf = io.BytesIO()

proc = subprocess.Popen(['ls', '-l'],
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
err_thread = threading.Thread(target=shutil.copyfileobj, 
    args=(proc.stderr, err_buf))
err_thread.start()
for line in proc.stdout:
    print(line.decode('utf-8'), end='')
retval = proc.wait()
err_thread.join()
print('error:', err_buf.getvalue())

Upvotes: 1

Related Questions