CODEWITHSUNDEEP
pythonloggingconfig
aether643
aether643

Reputation: 300

Python logging for multiple modules

I am working on a python project with multiple modules. I have tried to implements logging with a log.conf, which already works as expected. However, I am wondering, is there any other way to add logger into modules without adding loggers keys to log.conf? because it is quite cumbersome to add keys for each modules I need to add logger to.

Currently, my log.conf looks like this.

[loggers]
keys=root,main,xx_utils,xx_config,xx_extraction, ....

[handlers]
keys=consoleHandler,rotatingFileHandler

[formatters]
keys=defaultFormatters

[logger_root]
handlers=consoleHandler

[logger_main]
handlers=rotatingFileHandler
level=INFO
qualname=__main__
propagate=1

[xx_utils]
handlers=rotatingFileHandler
level=INFO
qualname=xx_utils
propagate=1
....

Essentially, I am wondering whether it is possible to creata a loggerUtils class that can be imported by each modules so that I don't need to add loggers keys to log.conf.

Upvotes: 0

Views: 1907

Answers (1)

bjk
bjk

Reputation: 110

As someone who has spent far too many hours learning how to do this the past few days, I think I'm finally there and will try to point you to what helped me.

This answer from the current logging maintainer explains how to implement with multiple modules very well: https://stackoverflow.com/a/15735146/2574074

And I found this answer very helpful as well: https://stackoverflow.com/a/4150322/2574074

To your specific problem, essentially, if all of your loggers have the same configuration (as it appears they do), then I think the easiest way for you would be to establish and configure your root logger in your config as follows:

[loggers]
keys=root

[handlers]
keys=consoleHandler,rotatingFileHandler

[formatters]
keys=defaultFormatters

[logger_root]
level=DEBUG
handlers=consoleHandler,rotatingFileHandler

[handler_consoleHandler]
... your consoleHandler configuration ...

[handler_rotatingFileHandler]
... your rotatingFileHandler configuration ...

...

You will not actually log to the root logger directly, however.

Instead, at the top of each of your modules, add this:

import logging
logger = logging.getLogger(__name__)

Or you could use whatever name you want:

import logging
logger = logging.getLogger(xx_utils)

This establishes a new logger for each module (i.e., no need to create a separate loggerUtils class) and because you are not setting level, each of these loggers will have level of NOTSET by default and so logs to these loggers will be passed to the parent logger(s), eventually reaching the root logger.

Important tip, which caused me trouble, is make sure you establish a logger for your main module in the same way as your other sub-modules, but be sure to do this after you've read in your config file and established/configured the root logger. Otherwise you'll automatically establish the root logger by calling logger = logging.getLogger(main) or logger = logging.getLogger(__name__) if running main.py as a script, but it will not have the configuration you want.

Upvotes: 1

Related Questions