Having two different handlers for logging in python logging module

Question

I am trying to have two different handlers where one handler will print the logs on console and other different handler will print the logs on console. Conslole handler is given by one inbuilt python modbus-tk library and I have written my own file handlers.

LOG = utils.create_logger(name="console", record_format="%(message)s") . ---> This is from modbus-tk library
LOG = utils.create_logger("console", level=logging.INFO)

logging.basicConfig(filename="log", level=logging.DEBUG)
log = logging.getLogger("simulator")
handler = RotatingFileHandler("log",maxBytes=5000,backupCount=1)
log.addHandler(handler)

What I need:

LOG.info("This will print message on console")

log.info("This will print message in file")

But problem is both the logs are getting printed on the console and both are going in file. I want only LOG to be printed on the console and log to be printed in the file.

edited:

Adding snippet from utils.create_logger

def create_logger(name="dummy", level=logging.DEBUG, record_format=None):
    """Create a logger according to the given settings"""
    if record_format is None:
        record_format = "%(asctime)s\t%(levelname)s\t%(module)s.%(funcName)s\t%(threadName)s\t%(message)s"

    logger = logging.getLogger("modbus_tk")
    logger.setLevel(level)
    formatter = logging.Formatter(record_format)
    if name == "udp":
        log_handler = LogitHandler(("127.0.0.1", 1975))
    elif name == "console":
        log_handler = ConsoleHandler()
    elif name == "dummy":
        log_handler = DummyHandler()
    else:
        raise Exception("Unknown handler %s" % name)
    log_handler.setFormatter(formatter)
    logger.addHandler(log_handler)
    return logger

Answer 1

There are a few problems in your code and without seeing the whole configuration it is hard to tell what exactly causes this, but most likely what is happening is that the logs are propagated.

First of all when you call basicConfig you are configuring the root logger and tell it to create a FileHandler with the filename log, but just two lines after that you are creating a RotatingFileHandler that uses the same file. Both loggers are writing to the same file now.

I find it always helps to understand the flow of how logging works in python: https://docs.python.org/3/howto/logging.html#logging-flow

And if you don't want all logs to be sent to the root logger too you should set LOG.propagate = False. That stops this logger from propagating their logs.

Answer 2

I have an own customized logging module. I have modified a little and I think now it can be proper for your problem. It is totally configurable and it can handle more different handlers.

If you want to combine the console and file logging, you only need to remove the return statement (I use this way).

I have written comment to code for more understandable and You can found a test section in if __name__ == "__main__": ... statement.

Code:

import logging
import os


# Custom logger class with multiple destinations
class CustomLogger(logging.Logger):
    """
    Customized Logger class from the original logging.Logger class.
    """

    # Format for console log
    FORMAT = (
        "[%(name)-30s][%(levelname)-19s] | %(message)-100s "
        "| (%(filename)s:%(lineno)d)"
    )

    # Format for log file
    LOG_FILE_FORMAT = "[%(name)s][%(levelname)s] | %(message)s " "| %(filename)s:%(lineno)d)"

    def __init__(
        self,
        name,
        log_file_path=None,
        console_level=logging.INFO,
        log_file_level=logging.DEBUG,
        log_file_open_format="w",
    ):

        logging.Logger.__init__(self, name)

        consol_color_formatter = logging.Formatter(self.FORMAT)

        # If the "log_file_path" parameter is provided,
        # the logs will be visible only in the log file.
        if log_file_path:
            fh_formatter = logging.Formatter(self.LOG_FILE_FORMAT)
            file_handler = logging.FileHandler(log_file_path, mode=log_file_open_format)
            file_handler.setLevel(log_file_level)
            file_handler.setFormatter(fh_formatter)
            self.addHandler(file_handler)
            return

        # If the "log_file_path" parameter is not provided,
        # the logs will be visible only in the console.
        console = logging.StreamHandler()
        console.setLevel(console_level)
        console.setFormatter(consol_color_formatter)
        self.addHandler(console)


if __name__ == "__main__":  # pragma: no cover
    current_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), "test_log.log")
    console_logger = CustomLogger(__file__, console_level=logging.INFO)

    file_logger = CustomLogger(__file__, log_file_path=current_dir, log_file_level=logging.DEBUG)

    console_logger.info("test_to_console")
    file_logger.info("test_to_file")

Console output:

>>> python3 test.py
[test.py][INFO               ] | test_to_console                                                                                      | (test.py:55)

Content of test_log.log file:

[test.py][INFO] | test_to_file | test.py:56)

If something is not clear of you have question/remark, let me know and I will try to help.

EDIT:

If you change the GetLogger to Logger in your implementation, it will work.

Code:

import logging


def create_logger(name="dummy", level=logging.DEBUG, record_format=None):
    """Create a logger according to the given settings"""
    if record_format is None:
        record_format = "%(asctime)s\t%(levelname)s\t%(module)s.%(funcName)s\t%(threadName)s\t%(message)s"

    logger = logging.Logger("modbus_tk")
    logger.setLevel(level)
    formatter = logging.Formatter(record_format)
    if name == "console":
        log_handler = logging.StreamHandler()
    else:
        raise Exception("Wrong type of handler")
    log_handler.setFormatter(formatter)
    logger.addHandler(log_handler)
    return logger


console_logger = create_logger(name="console")

# logging.basicConfig(filename="log", level=logging.DEBUG)
file_logger = logging.Logger("simulator")
handler = logging.FileHandler("log", "w")
file_logger.addHandler(handler)

console_logger.info("info to console")
file_logger.info("info to file")

Console output:

>>> python3 test.py
2019-12-16 13:10:45,963 INFO    test.<module>   MainThread      info to console

Content of log file:

info to file