CODEWITHSUNDEEP
pythonpython-3.xsubprocess
Shannon
Shannon

Reputation: 54

How to read stdout from python subprocess popen non-blockingly on Windows?

I am suffering from the Windows Python subprocess module.
This is test code1(named test1.py): 

import subprocess as sbp


with sbp.Popen('python tests/test2.py',stdout=sbp.PIPE) as proc:
    print('parent process')
    print(proc.stdout.read(1))
    print('end.')

and test code2(named test2.py): 

import random
import time

def r():
    while True:
        yield random.randint(0, 100)


for i in  r():
    print(i)
    time.sleep(1)

Generally, the test code2 generates random integer(0~100) and print it out infinitely. I want the test code1 create a subprocess and launch it, read the stdout in realtime(not waiting for subprocess finished). But when I run the code, the output is :

python.exe test1.py
parent process

It blocks on stdout.read() forever. I have tried:

  1. Replace stdout.read with communicate(), doesn't work as python doc expected, it will blocking until subprocess terminate.
  2. use poll() methods to detect subprocess and read n bytes, forever block on read()
  3. Modify the test2.code, only generate one nunber and break the loop. The father process print it out immediately(I think it's because child process terminated)

I searched a lot of similiar answers and did as they suggested(use stdout instead of communicate), but still didn't work?

Could anyone help me explaining why and how to do it?

This is my platform information:
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32

Upvotes: 1

Views: 6609

Answers (1)

Nikita
Nikita

Reputation: 6341

It has to do with Python's output buffering (for a child process in your case). Try disabling the buffering and your code should work. You can do it by either running python with -u key, or calling sys.stdout.flush().

To use the -u key you need to modify the argument in the call to Popen, to use the flush() call you need to modify the test2.py.

Also, your test1.py would print just a single number, because you read only 1 byte from the pipe, instead of reading them in a loop.

Solution 1:

test1.py

import subprocess as sbp

with sbp.Popen(["python3", "-u", "./test2.py"], stdout=sbp.PIPE) as proc:
    print("parent process")
    while proc.poll() is None:  # Check the the child process is still running
        data = proc.stdout.read(1)  # Note: it reads as binary, not text
        print(data)
    print("end")

This way you don't have to touch the test2.py at all.

Solution 2:

test1.py

import subprocess as sbp

with sbp.Popen("./test2.py", stdout=sbp.PIPE) as proc:
    print("parent process")
    while proc.poll() is None:  # Check the the child process is still running
        data = proc.stdout.read(1)  # Note: it reads as binary, not text
        print(data)
    print("end")

test2.py

import random
import time
import sys

def r():
    while True:
        yield random.randint(0, 100)

for i in  r():
    print(i)
    sys.stdout.flush()  # Here you force Python to instantly flush the buffer
    time.sleep(1)

This will print each received byte on a new line, e.g.:

parent process
b'9'
b'5'
b'\n'
b'2'
b'6'
b'\n'

You can switch the pipe to text mode by providing encoding in arguments or providing universal_newlines=True, which will make it use the default encoding. And then write directly to sys.stdout of your parent process. This will basically stream the output of a child process to the output of the parent process.

test1.py

import subprocess as sbp
import sys

with sbp.Popen("./test2.py", stdout=sbp.PIPE, universal_newlines=True) as proc:
    print("parent process")
    while proc.poll() is None:  # Check the the child process is still running
        data = proc.stdout.read(1)  # Note: it reads as binary, not text
        sys.stdout.write(data)
    print("end")

This will provide the output as if the test2.py is executed directly:

parent process
33
94
27

Upvotes: 4

Related Questions