CODEWITHSUNDEEP
pythonlogging
feeeper
feeeper

Reputation: 3017

Python TimedRotatingFileHandler overwrites logs

I setup TimedRotatingFileHandler like that:

import logging as logging
from logging.handlers import TimedRotatingFileHandler
import os
import time

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# new file every minute
rotation_logging_handler = TimedRotatingFileHandler('logs/log', 
                               when='m', 
                               interval=1, 
                               backupCount=5)
rotation_logging_handler.setLevel(logging.DEBUG)

format = u'%(asctime)s\t%(levelname)s\t%(filename)s:%(lineno)d\t%(message)s'
rotation_logging_handler.setFormatter(logging.Formatter(format))
rotation_logging_handler.suffix = '%Y-%m-%d'

logger.addHandler(rotation_logging_handler)

Usage:

logger.logging.info('Service started at port %s', config.get_param('port'))

while True:
    time.sleep(21)
    logger.logging.info('Now time is {}'.format(time.time()))

I expected that every minute new messages from logs/log had to append to existing log file for current date. Instead it every minute messages from logs/log overwrote existing log file for current date.

What should I do to reach that behaviour?

PS: After small research I found that TimedRotatingFileHandler in the doRollover method deletes existing log file and creates new file. So first solution is to create new handler derived from TimedRotatingFileHandler which creates new file (with some index for example) insted of deleting existing log file.

Upvotes: 3

Views: 22753

Answers (3)

Rohit
Rohit

Reputation: 4168

It is very much possible to change the filenames when they are rotated to anything you want by overrding the rotation_filename method of the BaseRotatingHandler class and setting it to an appropriate callable.

Here is a very trivial example of doing the same, but you can tweak it to suit your needs.

import logging
from logging.handlers import TimedRotatingFileHandler
import datetime as dt

def filer(self):
    now = dt.datetime.now()
    return 'file.txt'+now.strftime("%Y-%m-%d_%H:%M:%S")


logger = logging.getLogger()
rotating_file_handler = TimedRotatingFileHandler(filename="/Users/rbhanot/file.txt",
                                                 when='S',
                                                 interval=2,
                                                 backupCount=5)
rotating_file_handler.rotation_filename = filer
formatter = logging.Formatter(
    '%(asctime)s %(name)s:%(levelname)s - %(message)s')
rotating_file_handler.setFormatter(formatter)
logger.addHandler(rotating_file_handler)
logger.setLevel(logging.DEBUG)

logger.info("hello")

Here is the output of

❯ ls file*
file.txt  file.txt2020-10-06_13:12:13  file.txt2020-10-06_13:12:15  file.txt2020-10-06_13:13:45

Upvotes: 6

feeeper
feeeper

Reputation: 3017

After little bit more researching I found BaseRotatingHandler.namer attribute usage in the BaseRotatingHandler.rotation_filename method:

The default implementation calls the 'namer' attribute of the handler, if it's callable, passing the default name to it. If the attribute isn't callable (the default is None), the name is returned unchanged.

So as a solution I implemented my own namer function that got filename and returned new filename with my template:

  • 20181231.log
  • 20181231.0.log
  • 20181231.1.log
  • etc.

Full example:

import logging as logging
from logging.handlers import TimedRotatingFileHandler
import os
import time


def get_filename(filename):
    # Get logs directory
    log_directory = os.path.split(filename)[0]

    # Get file extension (also it's a suffix's value (i.e. ".20181231")) without dot
    date = os.path.splitext(filename)[1][1:]

    # Create new file name
    filename = os.path.join(log_directory, date)

    # I don't want to add index if only one log file will exists for date
    if not os.path.exists('{}.log'.format(filename)):
        return '{}.log'.format(filename)

    # Create new file name with index
    index = 0
    f = '{}.{}.log'.format(filename, index)
    while os.path.exists(f):
        index += 1
        f = '{}.{}.log'.format(filename, index)
    return f


format = u'%(asctime)s\t%(levelname)s\t%(filename)s:%(lineno)d\t%(message)s'
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# new file every minute
rotation_logging_handler = TimedRotatingFileHandler('logs/log', 
                               when='m', 
                               interval=1, 
                               backupCount=5)
rotation_logging_handler.setLevel(logging.DEBUG)
rotation_logging_handler.setFormatter(logging.Formatter(format))
rotation_logging_handler.suffix = '%Y%m%d'
rotation_logging_handler.namer = get_filename

logger.addHandler(rotation_logging_handler)

Upvotes: 1

Giacomo Alzetta
Giacomo Alzetta

Reputation: 2479

According to the documentation of TimedRotatingFileHandler:

The system will save old log files by appending extensions to the filename. The extensions are date-and-time based, using the strftime format %Y-%m-%d_%H-%M-%S or a leading portion thereof, depending on the rollover interval.

In other words: by modifying suffix you are breaking the rollover. Just leave it at default and python will create the files called:

  • logs/log2018-02-02-01-30
  • logs/log2018-02-02-01-31
  • logs/log2018-02-02-01-32
  • logs/log2018-02-02-01-33
  • logs/log2018-02-02-01-34

And after this (if backupCount=5) it will delete the -30 one and create the -35.

If you instead want to have names like :

  • logs/log2018-02-02-01.0
  • logs/log2018-02-02-01.1
  • logs/log2018-02-02-01.2
  • logs/log2018-02-02-01.3
  • logs/log2018-02-02-01.4

Where 0 is the newest one and .4 is the oldest one, then yes, that handler has not been designed to do that.

Upvotes: 2

Related Questions