CODEWITHSUNDEEP
pythonpython-3.xstdout
martync
martync

Reputation: 360

Capturing output stream from python 3

I'm trying to capture the output stream of executed commands from a python script. All is going well until the command is a python command.

Here are my functions that get the streams with different callbacks on stdout and stderr :

capture_output.py

import sys
import logging
import shlex
import asyncio


async def _read_stream(stream, cb):  
    while True:
        line = await stream.readline()
        if line:
            cb(line.decode())
        else:
            break


async def _stream_subprocess(cmd, stdout_cb, stderr_cb):  
    process = await asyncio.create_subprocess_exec(*cmd,
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)

    await asyncio.wait([
        _read_stream(process.stdout, stdout_cb),
        _read_stream(process.stderr, stderr_cb)
    ])
    return await process.wait()


def execute(cmd, stdout_cb, stderr_cb):  
    loop = asyncio.get_event_loop()
    rc = loop.run_until_complete(
        _stream_subprocess(
            cmd,
            stdout_cb,
            stderr_cb,
    ))
    loop.close()
    return rc


def run_command(command,
                output_function=lambda x: print("STDOUT: %s" % x),
                error_handling=lambda x: print("STDERR: %s" % x),
                cwd=None):
    execute(shlex.split(command), output_function, error_handling, )


if __name__ == "__main__":
    "Get the command on 1st position and run it"
    command_str = sys.argv[1]
    run_command(command_str)

Here is an example of external command :

anylongrunning_script.sh

#!/bin/bash
for i in {1..5}
do
   echo "$i"
   sleep 0.5
done

If I run python capture_output.py "sh anylongrunning_script.sh", I see the live stream, everything is going well.

But if run a python command like this python capture_output.py "python anylongrunning_script.py" :

anylongrunning_script.py

import time

for i in range(1, 6):
    print(i)
    time.sleep(0.5)

The output will be rendered in 1 block at the end. Do you know what is the difference between a shell "echo" and a python "print" in this case ?

Upvotes: 0

Views: 1607

Answers (1)

avigil
avigil

Reputation: 2246

Disable output buffering by running your command with the -u option. Try python capture_output.py "python -u anylongrunning_script.py"

You can make sure all python processes use unbuffered mode by setting the PYTHONUNBUFFERED env variable for your subprocesses.

Upvotes: 1

Related Questions