CODEWITHSUNDEEP
pythonpython-3.xsubprocess
IdecEddy
IdecEddy

Reputation: 159

Python subprocess.poll() questions

I am working with Pythons subprocess module and and having issues polling a running process.

My end goal is to grab each new line from stdout evaluate it printing out relevant new lines the main process's stdout as it runs. The issues is that when I poll a test program it seems to wait for the child process to finish before outputting anything to the stdout. It is doing this when I introduce a time.sleep(1) method to the parent program.

Parent Program

import subprocess
import time
def runProcess():
    process = subprocess.Popen([
        'py.exe',
        'subpy.py'],
        stdin=subprocess.PIPE,
        stderr=subprocess.PIPE,
        stdout=subprocess.PIPE)
    while process.poll() == None:
        print('.', end ="")
        time.sleep(1)

runProcess()
input("press enter to exit.")

Child Program

import random, time

for x in range(1,5):
    time.sleep(1)
    value = random.randint(1,3)
    if value == 1: 
        print("hello")
    if value == 2:
        print('goodbye')
    if value == 3:
        print('im not sure')
exit()

Any help would be appreciated thanks!

PS:

My end goal for the parent process would look something like this, although i'm not sure if this would run into issues of polling before an output is totally completed.

import subprocess
import time
def runProcess():
    process = subprocess.Popen([
        'py.exe',
        'subpy.py'],
        stdin=subprocess.PIPE,
        stderr=subprocess.PIPE,
        stdout=subprocess.PIPE)
    while process.poll() == None:
        stdout, stderr = subprocess.communicate()
        if stdout == '':
            print('.', end ="")
            time.sleep(1)
        if stdout == "goodbye" or stdout == 'hello'
            print(">>> %s" % stdout)
        else:
            print(stdout)

runProcess()
input("press enter to exit.")

Upvotes: 1

Views: 3502

Answers (1)

tdelaney
tdelaney

Reputation: 77407

Once you create the process, you can read its stdout until it closes, indicating that the process has closed. But there are a couple of problems.

The first is that the process may fill up stderr and block trying to write more. That is solved with a background thread that reads stderr for you. In this example, I just copy it to an in-memory buffer to read after the process exits. There are other options, depending on what you want to do with the data stream.

Then there's the question of how often the stdout pipe is flushed. Since its a pipe, writing is block buffered. Without flushes coming from the subprocess, you won't get the output real time. In unix-like systems, you can replace the pipe with a pseudo-tty (see pty module). But this is Windows, so there isn't much you can do from the calling process. What ends up happening is that you get the incoming lines in groups based on when the clibrary of the child flushes (or you put a lot of flushes in the code).

import subprocess
import sys
import io
import time
import shutil
import threading

def runProcess():
    process = subprocess.Popen([
        sys.executable,
        'subpy.py'],
        stdin=subprocess.PIPE,
        stderr=subprocess.PIPE,
        stdout=subprocess.PIPE)
    process.stdin.close()
    err_buf = io.BytesIO()
    err_thread = threading.Thread(target=shutil.copyfileobj,
            args=(process.stderr, err_buf))
    err_thread.start()
    for line in process.stdout:
        line = line.decode() # defaulting to system encoding
        print(line,end='',flush=True)
    process.wait()
    err_thread.join()
    err_buf.seek(0)
    print("Errors:", err_buf.read().decode())

runProcess()
input("press enter to exit.")

Upvotes: 1

Related Questions