Redirect STDOUT and STDERR to python logger and also to jupyter notebook

Question

Important to know: I am working on jupyter notebook.

I want to create a logger to which I will redirect the STDOUT and STDERR but I also want to see those outputs on the jupyter notebook output console.

So far what I have implemented is:

import logging
import sys

class StreamToLogger(object):
    """
    Fake file-like stream object that redirects writes to a logger instance.
    """
    def __init__(self, logger, log_level=logging.INFO):
        self.logger = logger
        self.log_level = log_level
        self.linebuf = ''

    def write(self, buf):
        for line in buf.rstrip().splitlines():
            self.logger.log(self.log_level, line.rstrip())

    def flush(self):
        pass

logging.basicConfig(filename='my_log.log',
                    filemode='a',
                    # stream=sys.stdout,
                    level=logging.DEBUG,
                    format='%(asctime)s;%(name)s;%(levelname)s;%(message)s')

# redirect stdout and stderr to logger
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl

stderr_logger = logging.getLogger('STDERR')
s2 = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = s2

log = logging.getLogger('my_log')

# An info log
log.info('This is an info')

# A stdout message
print("This is an STDOUT")

# A stderr message
1 / 0

First Question: The previous code, if stored on a .py file, is able to redirect the stdout and stderr to the my_log.log file. But I lose track of the messages in a normal terminal console.

What I would like is to have both the stderr and stdout redirected to the log file and also be able to see them on the console.

Second Question: I am working on jupyter notebooks and I would like to be able to log from there. This means all the stdout and stderr redirected from jupyter notebook output to the log file, but also keep it on the jupyter notebook console. I realized that the code above, redirects the stdout to the log file but not the stderr, and as a result all my prints('XX') are in my log file and my exceptions are still on the notebook console.

Seems like jupyter notebooks deals in a different way with STDOUT and STDERR

Thanks for your help

Answer 1

It might be a little late.
I had similar demands and the best way I can find to do it is something similar to the following code. I use it to track outputs of model training that will run for days. It can log to a file by calling Std2File.enable(file_name) and rollback by calling Std2File.disable(). It can log to the file and show it to the notebook. the only shortness is that the output displayed on the notebook will always show in the cell of calling enable.

class Std2File(object):
    """
    Redirect stoout and stderr to a local file.
    It is designed for use case of notebook where after you close a notebook you can record the stdout and stderr
    """
    stdout = None
    stderr = None
    fd = None

    def __init__(self, f, std):
        self.std = std
        self.f = f

    @staticmethod
    def enable(f='/tmp/std_copy.log'):
        if Std2File.stdout is None:
            Std2File.stdout = sys.stdout
            Std2File.stderr = sys.stderr
            Std2File.fd = open(f, 'a+')
            sys.stdout = Std2File(Std2File.fd, sys.stdout)
            sys.stderr = Std2File(Std2File.fd, sys.stderr)
        print('\n\nbackup stdout/stderr to %s at %s\n' % (f, dt.datetime.now()))

    @staticmethod
    def disable():
        if Std2File.stdout is not None:
            sys.stdout = Std2File.stdout
            sys.stderr = Std2File.stderr
            Std2File.stderr = Std2File.stdout = None
            Std2File.fd.close()
            Std2File.fd = None

    def __getattr__(self, name):
        return object.__getattribute__(self.f, name)

    def write(self, x):
        self.std.write(x)
        self.std.flush()
        self.f.write(x)
        self.f.flush()

    def flush(self):
        self.std.flush()
        self.f.flush()