CODEWITHSUNDEEP
pythonloggingmodule
John Murray
John Murray

Reputation: 13

Python logging multiple modules each module log to separate files

I have a main script and multiple modules. Right now I have logging setup where all the logging from all modules go into the same log file. It gets hard to debug when its all in one file. So I would like to separate each module into its own log file. I would also like to see the requests module each module uses into the log of the module that used it. I dont know if this is even possible. I searched everywhere and tried everything I could think of to do it but it always comes back to logging everything into one file or setup logging in each module and from my main module initiate the script instead of import as a module.

main.py

import logging, logging.handlers
import other_script.py

console_debug = True
log = logging.getLogger()

def setup_logging():    
    filelog = logging.handlers.TimedRotatingFileHandler(path+'logs/api/api.log',
                when='midnight', interval=1, backupCount=3)
    filelog.setLevel(logging.DEBUG)
    fileformatter = logging.Formatter('%(asctime)s %(name)-15s %(levelname)-8s %(message)s')
    filelog.setFormatter(fileformatter)
    log.addHandler(filelog)

    if console_debug:
        console = logging.StreamHandler()
        console.setLevel(logging.DEBUG)
        formatter = logging.Formatter('%(name)-15s: %(levelname)-8s %(message)s')
        console.setFormatter(formatter)
        log.addHandler(console)

if __name__ == '__main__':
    setup_logging()

other_script.py

import requests
import logging

log = logging.getLogger(__name__)

Upvotes: 0

Views: 2228

Answers (1)

blues
blues

Reputation: 5185

One very basic concept of python logging is that every file, stream or other place that logs go is equivalent to one Handler. So if you want every module to log to a different file you will have to give every module it's own handler. This can also be done from a central place. In your main.py you could add this to make the other_script module log to a separate file:

other_logger = logging.getLogger('other_script')
other_logger.addHandler(logging.FileHandler('other_file'))
other_logger.propagate = False

The last line is only required if you add a handler to the root logger. If you keep propagate at the default True you will have all logs be sent to the root loggers handlers too. In your scenario it might be better to not even use the root logger at all, and use a specific named logger like getLogger('__main__') in main.

Upvotes: 2

Related Questions