CODEWITHSUNDEEP
pythonsubprocesspopen
Nick
Nick

Reputation: 675

Read subprocess stdout while maintaining it in the buffer

Is it possible to read a subprocess's stdout but at the end of the program still maintain the whole process.communicate()?

For example i have a python script that starts a c# app in a Popen subprocess and checks the log file it produces live to determine the state it is in but certain errors are not dumped in the logs and are in the stdout and certain errors there will cause the process to hang. I want to be able to detect that output and in those cases be able to kill the process.

This is easy while doing process.stdout.readline() but at the end of the program i loose the ability to dump the entire stdout in it's native formatting because the buffer get's flushed after reading. Is it possible to maintain that buffer?

Or at the very least save the read lines in a variable in it's native formatting?

Upvotes: 0

Views: 1433

Answers (1)

tdelaney
tdelaney

Reputation: 77337

You can read stdout line by line, process it and save it to a list or buffer, and have the buffer available later. In this example processing is just print, but you could change that however you want. I also assumed you just want to collect stderr in the background, so created a separate thread.

import subprocess as subp
import threading
import io

def _pipe_read_thread(stream, output):
    output.write(stream.read())
    stream.close()

def proc_runner(cmd):
    stdout_lines = []
    stdout_buf = io.BytesIO()
    stderr_buf = io.BytesIO()
    p = subp.Popen(cmd, stdout=subp.PIPE, stderr=subp.PIPE)
    stderr_t = threading.Thread(target=_pipe_read_thread,
        args=(p.stderr, stderr_buf))
    stderr_t.start()
    for line in p.stdout:
        print(line)
        stdout_buf.write(line)
    returncode = p.wait()
    stderr_t.join()
    stdout_buf. seek(0)
    stderr_buf.seek(0)
    return returncode, stdout_buf, stderr_buf

returncode, stdout, stderr = proc_runner(['ls', '-a'])
print('=============================')
print(stdout.read())
print('=============================')
print(stderr.read())

Upvotes: 2

Related Questions