CODEWITHSUNDEEP
pythonconsolesubprocesstracesys
Chris
Chris

Reputation: 5804

How to run a python script while storing live output from the console?

I'd like to create a function that executes python scripts while storing the console output live as it's executing.

For example, I use the subprocess module to run example.py, but I only receive the console output after the entire script has run instead of getting the console output as it's happening. In other words and in accordance with the below script, I'd like to receive the console output "hello world" immediately, and then wait 60 seconds, and then receive the console output "goodbye world"

example.py

import time 

print "hello world!"
time.sleep(60)
print "goodbye world"

Below is the script that runs the script in example.py and stores the console after

import subprocess
script = open('example.py',"r+").read()
process = subprocess.Popen(['python', '-'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.stdin.write(script)
stored_console_output, stored_console_output_error = process.communicate()
print stored_console_output

This returns the following as a string after the entire script has executed

hello world!
goodbye world

Note: I cannot change the python script example.py. I can only change the function that calls it.

In addition to getting the console output live (if possible), I'd like to to get the python line that led to that console output. For example, I'd like to attain the following 

import time 

print "hello world!"
hello world
time.sleep(60)
print "goodbye world"
goodbye world

I've also tried to utilize sys module, but it doesn't store the console output: 

import sys
import inspect

class SetTrace(object):
    def __init__(self, func):
        self.func = func

    def __enter__(self):
        sys.settrace(self.func)
        return self

    def __exit__(self, ext_type, exc_value, traceback):
        sys.settrace(None)

def monitor(frame, event, arg):
    if event == "line":
        print event
    return monitor


with SetTrace(monitor):
    exec(open('example.py',"r+").read())

This returns the following and it does it live. 

line
line
line
hello world!
line
line
goodbye world
line

Upvotes: 2

Views: 1622

Answers (1)

andrew_reece
andrew_reece

Reputation: 21264

This post largely answers your question, although there's one comment in particular that provides the key to your specific problem: you need the -u flag when calling example.py to prevent STDOUT buffering on sleep().

Borrowing heavily from the aforementioned answer, this solution works:

from subprocess import Popen, PIPE

def execute(cmd):
    popen = Popen(cmd, stdout=PIPE, universal_newlines=True)
    for stdout_line in iter(popen.stdout.readline, ""):
        yield stdout_line 
    popen.stdout.close()

for statement in execute(['python', '-u', 'example.py']):
    print(statement, end="")

Output:

Hello
# pauses for the number of sleep seconds
Goodbye

Upvotes: 2

Related Questions