IPython: redirecting output of a Python script to a file (like bash >)

Question

I have a Python script that I want to run in IPython. I want to redirect (write) the output to a file, similar to:

python my_script.py > my_output.txt

How do I do this when I run the script in IPython, i.e. like execfile('my_script.py')

There is an older page describing a function that could be written to do this, but I believe that there is now a built-in way to do this that I just can't find.

Answer 1

I wonder why the verified solution doesn't entirely work in a loop, the following:

for i in range(N):
    with redirect_output("path_to_output_file"):
        %run <python_script> arg1 arg2 arg3

creates N files in the directory with only output from the first print statement of the <python_script>. Just confirming- when run separately for each iteration of the for loop, the script produces the right result.

Answer 2

While this an old question, I found this and the answers as I was facing a similar problem.

The solution I found after sifting through IPython Cell magics documentation is actually fairly simple. At the most basic the solution is to assign the output of the command to a variable.

This simple two-cell example shows how to do that. In the first Notebook cell we define the Python script with some output to stdout making use of the %%writefile cell magic.

%%writefile output.py
print("This is the output that is supposed to go to a file")

Then we run that script like it was run from a shell using the ! operator.

output = !python output.py
print(output)
>>> ['This is the output that is supposed to go to a file']

Then you can easily make use of the %store magic to persist the output.

%store output >output.log

Notice however that the output of the command is persisted as a list of lines. You might want to call "\n".join(output) prior storing the output.

Answer 3

To quickly store text contained in a variable while working in IPython use %store with > or >>:

%store VARIABLE >>file.txt (appends)
%store VARIABLE >file.txt (overwrites)

(Make sure there is no space immediately following the > or >>)

Answer 4

For just one script to run I would do the redirection in bash

ipython -c "execfile('my_script.py')" > my_output.txt

On python 3, execfile does not exist any more, so use this instead

ipython -c "exec(open('my_script.py').read())" > my_output.txt

Be careful with the double vs single quotes.

Answer 5

use this code to save the output to file

import time
from threading import Thread
import sys
#write the stdout to file
def log():
    #for stop the thread
    global run
    while (run):
        try:
            global out
            text = str(sys.stdout.getvalue())
            with open("out.txt", 'w') as f:
                f.write(text)
        finally:
            time.sleep(1)

%%capture out
run = True
print("start")
process = Thread(target=log, args=[]).start()

# do some work
for i in range(10, 1000):
    print(i)
    time.sleep(1)
run= False
process.join()

It is useful to use a text editor that tracer changes the file and suggest reloading the file like notepad++

Answer 6

It seems a lot of code.... My solution. redirect output of ipython script into a csv or text file like sqlplus spool wonder there is an easy way like oracle sqlplus spool command..?

Answer 7

IPython has its own context manager for capturing stdout/err, but it doesn't redirect to files, it redirects to an object:

from IPython.utils import io
with io.capture_output() as captured:
    %run my_script.py

print captured.stdout # prints stdout from your script

And this functionality is exposed in a %%capture cell-magic, as illustrated in the Cell Magics example notebook.

It's a simple context manager, so you can write your own version that would redirect to files:

class redirect_output(object):
    """context manager for reditrecting stdout/err to files"""


    def __init__(self, stdout='', stderr=''):
        self.stdout = stdout
        self.stderr = stderr

    def __enter__(self):
        self.sys_stdout = sys.stdout
        self.sys_stderr = sys.stderr

        if self.stdout:
            sys.stdout = open(self.stdout, 'w')
        if self.stderr:
            if self.stderr == self.stdout:
                sys.stderr = sys.stdout
            else:
                sys.stderr = open(self.stderr, 'w')

    def __exit__(self, exc_type, exc_value, traceback):
        sys.stdout = self.sys_stdout
        sys.stderr = self.sys_stderr

which you would invoke with:

with redirect_output("my_output.txt"):
    %run my_script.py

Answer 8

There's the hacky way of overwriting sys.stdout and sys.stderr with a file object, but that's really not a good way to go about it. Really, if you want to control where the output goes from inside python, you need to implement some sort of logging and/or output handling system that you can configure via the command line or function arguments instead of using print statements.