Reputation: 14553
I have a program with a main script (main.py) and a few modules (mod1.py, mod2.py, etc.).
In main.py, I configured a root logger to write messages in my terminal and in a file.
# main.py
import logging
LOG = logging.getLogger()
def init_logger():
"""..."""
if __name__ == '__main__':
init_logger()
LOG.info("...")
And in other modules, I use the standard idiom:
# mod1.py
import logging
LOG = logging.getLogger(__name__)
All is working fine and for each line in my log file, I know which module it originated from. For example, my log file contains something like:
(...)
13:52:59 [INFO |root] (MainThread) Logger initialized (/some_path/logs/log.txt)
13:53:00 [DEBUG|mod1] (MainThread) Some message
13:53:01 [INFO |mod234] (thread_1) Some other message
(...)
But now, I wish to log some very specific messages in a separate file. I don't want it to pollute my existing log file and I still want to know messages origins (I guess I have to keep a root logger).
ie. I don't want to filter by level (debug, info, error, etc.), but I want to filter by some kind of category.
How do I achieve that ?
Just for the record, my init_logger function, but I'm not sure it's relevant:
LOG = logging.getLogger()
def init_logger():
logger = logging.getLogger()
stream_formatter = logging.Formatter("%(asctime)s :: %(levelname)s :: %(message)s")
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(stream_formatter)
logger.addHandler(stream_handler)
log_path = os.path.join(os.path.dirname(__file__), "logs", "log.txt")
file_formatter = logging.Formatter(fmt="%(asctime)s [%(levelname)-5s|%(name)s] (%(threadName)-10s) %(message)s",
datefmt="%H:%M:%S")
file_handler = logging.FileHandler(log_path, mode='w')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(file_formatter)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG)
for tp_module in ("easyprocess",
"pyscreenshot",
"PIL"):
logging.getLogger(tp_module).setLevel(logging.CRITICAL)
logger.info("Main logger initialized (%s)", log_path)
init_logger()
LOG.info("...")
Upvotes: 0
Views: 4056
Reputation: 25809
If you want to keep the same structure, you can create two file loggers and assign a different filter to each of them. The root logger would filter out the very specific messages while the other logger would only consider those same messages. What matters the most, then, is based on what would you want to filter. Assuming it's the content, it would be as simple as creating a filter:
import logging
class ContentFilter(logging.Filter):
def __init__(self, content, name="", exclude=False):
super(ContentFilter, self).__init__(name)
self.content = content
self.exclude = exclude # allow for exclusion behavior as well
def filter(self, record):
if record.msg: # ignore empty logs
if self.content in record.msg: # note: you can filter on other `record` fields...
return not self.exclude
return self.exclude
Then when you're setting up your logging make sure that one file logger has a ContentFilter
set to filter in, while the other has it set to filter out and everything should work just fine. For example:
log = logging.getLogger() # init the root logger
log.setLevel(logging.DEBUG) # make sure we capture all levels for our test
# make a formatter to use for the file logs, shortened version from yours...
formatter = logging.Formatter("%(asctime)s [%(levelname)-5s|%(name)s] %(message)s")
# first file handler:
handler = logging.FileHandler("log1.log") # create a 'log1.log' handler
handler.setLevel(logging.DEBUG) # make sure all levels go to it
handler.setFormatter(formatter) # use the above formatter
handler.addFilter(ContentFilter("special content", exclude=True)) # exclude 'special content'
log.addHandler(handler) # add the file handler to the root logger
# second file handler:
handler = logging.FileHandler("log2.log") # create a 'log2.log' handler
handler.setLevel(logging.DEBUG) # make sure all levels go to it
handler.setFormatter(formatter) # use the above formatter
handler.addFilter(ContentFilter("special content")) # include only 'special content'
log.addHandler(handler) # add the file handler to the root logger
And once set up, you can test it as:
# first create a few fake 'module' loggers for a good measure:
log1 = logging.getLogger("module1")
log2 = logging.getLogger("module2")
log3 = logging.getLogger("module3")
log.info("Root logger without that content...")
log.info("Root logger with the special content...")
log1.info("Module1 logger without that content...")
log1.info("Module1 logger with the special content...")
log2.info("Module2 logger without that content...")
log2.info("Module2 logger with the special content...")
log3.info("Module3 logger without that content...")
log3.info("Module3 logger with the special content...")
Which would result in two files being created, log1.log
containing:
2018-07-17 16:21:19,780 [INFO |root] Root logger without that content... 2018-07-17 16:21:19,781 [INFO |module1] Module1 logger without that content... 2018-07-17 16:21:19,781 [INFO |module2] Module2 logger without that content... 2018-07-17 16:21:19,782 [INFO |module3] Module3 logger without that content...
And log2.log
containing:
2018-07-17 16:21:19,781 [INFO |root] Root logger with the special content... 2018-07-17 16:21:19,781 [INFO |module1] Module1 logger with the special content... 2018-07-17 16:21:19,781 [INFO |module2] Module2 logger with the special content... 2018-07-17 16:21:19,782 [INFO |module3] Module3 logger with the special content...
Effectively redirecting individual log messages to different files based on their content (or lack thereof).
Upvotes: 4