CODEWITHSUNDEEP
pythonlogging
Mikkel Jans
Mikkel Jans

Reputation: 125

logging in continouous loop

What would be a good way to create logs (with python logging module) inside a constant running loop, without producing a large amount of useless log-files? An example would be a loop that constant list a folder, and does some action when it sees a file of a specific type. I want to log that no files were found, or files were found but of a wrong type, without logging that same line constantly for each folder check, as it might run many times a second.

Upvotes: 5

Views: 6916

Answers (5)

Kound
Kound

Reputation: 2531

A bit of a hack but much easier is "misusing" functools lru_cache.

from functools import lru_cache
from logging import getLogger

# Keep track of 10 different messages and then warn again
@lru_cache(10)
def warn_once(logger: Logger, msg: str):
    logger.warning(msg)

You can increase the 10 to suppress more if required or set it to None to store suppress everything duplicated.

Upvotes: 0

Matt Joiner
Matt Joiner

Reputation: 118540

Create a Handler that subclasses whatever other functionality you need. Store either the last, or all the previously logged messages that you don't want to emit again:

def make_filetype_aware_handler(handler_class):
    class DontRepeatFiletypeHandler(handler_class):

        def __init__(self, *args, **kwds):
            super().__init__(*args, **kwds)
            self.previous_types = set()

        def emit(self, record):
            if not record.file_type in self.previous_types:
                self.previous_types.add(record.file_type)
                super().emit(record)
    return DontRepeatFiletypeHandler

FiletypeStreamHandler = make_filetype_aware_handler(logging.StreamHandler)

logger = logging.getLogger()
logger.addHandler(FiletypeStreamHandler(sys.stderr))
logger.debug('Found file of type %(file_type)', file_type='x-type/zomg')

Upvotes: 4

Nix
Nix

Reputation: 58522

My understanding is that you are trying to limit logging the same message over and over again.

If this is your issue, I would create a a set of file_types you have already logged. However you need to be careful, if this is going to run forever, you will eventually crash..

from sets import Set
logged = Set()
while yourCondition:
   file_type = get_next_file_type()
   needToLog = #determine if you need to log this thing

   if needToLog and (not file_type in logged):
      logger.info("BAH! " + file_type)
      logged.add(file_type)

Upvotes: 0

Paulo Scardine
Paulo Scardine

Reputation: 77271

Log the less important events with a lower precedence, like DEBUG. See setLevel and SysLogHandler.

At development time set the level to DEBUG, and as your application matures, set it to more reasonable values like INFO or ERROR.

Your app should do something about the errors, like remove files of the wrong type and/or create lacking files; or move the wrongfully configured directories from the job polling to a quarantine location, so your log will not be flooded.

Upvotes: 0

sam
sam

Reputation: 19164

import logging
logger = logging.getLogger(test)
# logging to a file
hdlr = logging.FileHandler(test.log)
formatter = logging.Formatter('%(asctime)s %(filename)s %(lineno)s %(levelname)s %    (message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr) 
logger.setLevel(logging.DEBUG)

Then in the loop, you have to check for file type and if file is present or not. Then add : 

logger.debug('File Type : %s ' % file_type)

also 

if file_present:
    logger.debug('File Present : %s ' % present)

Upvotes: 0

Related Questions