How to wildly and/or alternatingly read from stdout and/or stderr and write to stdin of a process in python?

Question

I searched here a lot of different questions and answers, but I did not found a general approach for:

reading from stdout and stderr, what is available - until the last byte (currently!) available (even if it is not a )
depending upon the read information write something to stdin
the command line tool will react to this stdin input and write (much later) something/nothing
start from beginning - or leave loop if process has finished, capturing its return code

Most examples here findable write only ONCE to stdin and read only ONCE (before/afterwards) from stdout and/or stderr. My intention is to "weave" reading from stdout and/or stderr and writing to stdin! Here an example:

starting a command line tool (finally with parameters) - e. g. python3.exe
reading always from the stdout and stderr
e. g. read everything and after reading >>> from stdout
write print('Hello World.')
e. g. read everything (Hello World.)and after reading >>> from stdout
write x = [6, 0]
e. g. read everything and after reading >>> from stdout
write y = x[0] / x[1]
e. g. read everything ( ... ZeroDivisionError: division by zero on stdout/stderr)
...

I tried to solve it with this found internet example (after other failed attempts):

# Example #27
# of https://www.programcreek.com/python/example/85342/asyncio.create_subprocess_shell
# Source Project: Python-Journey-from-Novice-to-Expert   Author: PacktPublishing   File: 07_processes.py    License: MIT License    5 votes vote downvote up
import asyncio
import sys

async def read_from_pipe(pipe, buf, timeout_sec):
    while True:
        try:
            pipe_byte = await asyncio.wait_for(pipe.read(1), timeout_sec)
        except asyncio.TimeoutError:
            break
        else:
            if len(pipe_byte) == 1:
                buf.append(pipe_byte[0])
            else:
                pipe_byte == b'
' # in case of end of file: fake end of line 
            if pipe_byte == b'
':
                return len(buf)

async def run_script(version):
    process = await asyncio.create_subprocess_shell(
        r'C:\Programs\Python\Python38-32\python.exe',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.PIPE,
    )

    if version == 0:
        # Write a simple Python script to the interpreter
        process.stdin.write(b'
'.join((
            b'import math',
            b'x = 2 ** 8',
            b'y = math.sqrt(x)',
            b'z = math.sqrt(y)',
            b'print("x: %d" % x)',
            b'print("y: %d" % y)',
            b'print("z: %d" % z)',
            b'for i in range(int(z)):',
            b'    print("i: %d" % i)',
        )))
        # Make sure the stdin is flushed asynchronously
        await process.stdin.drain()
        # And send the end of file so the Python interpreter will
        # start processing the input. Without this the process will
        # stall forever.
        process.stdin.write_eof()

        # Fetch the lines from the stdout asynchronously
        async for out in process.stdout:
            # Decode the output from bytes and strip the whitespace
            # (newline) at the right
            print(out.decode('utf-8').rstrip())

        # Wait for the process to exit
        await process.wait() 
    elif version == 1:
        cmds = [b'import math',
          b'x = 2 ** 8',
          b'y = math.sqrt(x)',
          b'z = math.sqrt(y)',
          # b'q = z / 0',
          b'print("x: %d" % x)',
          b'print("y: %d" % y)',
          b'print("z: %d" % z)',
          b'for i in range(int(z)):',
          b'    print("i: %d" % i)',
          b'exit(0)',
        ]
        idx = 0
        while True:
            stdout_buf = bytearray(b'')           
            out_read = await read_from_pipe(process.stdout, stdout_buf, 0.5)
            print(f'stdout[{out_read}]: {stdout_buf.decode("ascii")}
') if out_read else None
            stderr_buf = bytearray(b'')             
            err_read = await read_from_pipe(process.stderr, stderr_buf, 0.5)
            print(f'stderr[{err_read}]: {stdout_buf.decode("ascii")}
') if err_read else None
            if idx < len(cmds):
                current_cmd = cmds[idx].decode('ascii')
                print(f'writing command at index {idx}: "{current_cmd}"')
                process.stdin.write(cmds[idx])
                process.stdin.write(b'
')
                await process.stdin.drain()
                process.stdin.write_eof() # tried with/without this line, afterwards program hangs
                idx += 1
            else:
                break
        await process.wait()             

if sys.platform == "win32":
    codepage = 'cp437'
    loop = asyncio.ProactorEventLoop() # For subprocess' pipes on Windows
    asyncio.set_event_loop(loop)
else:
    codepage = 'utf-8'
    loop = asyncio.get_event_loop()

version = 1 # version = 0 runs but is not alternatingly reading stdout/stderr and writing to stdin!
returncode = loop.run_until_complete(run_script(1))
print(f'done with return code = {returncode}.')

Currently it doesn't read anything from the stdout and stderr. And after the entries in cmds are written, program hangs too. Finally it should run under linux.

How do I write the program correctly?

Is python3.exe a "too special" command line tool and is the root cause of these problems?

Hint: This example and the solution do not have to be performant at all. The intended command line tool to control is quite slow (overall execution 20 s to 20 min). Multithreading and multiprocessing is not really required, if not needed for a (simplified) working solution.

How to wildly and/or alternatingly read from stdout and/or stderr and write to stdin of a process in python?

Answers (1)

Related Questions