CODEWITHSUNDEEP
pythondjangopython-3.xdjango-rest-frameworkpython-logging
hardik24
hardik24

Reputation: 1048

Django logging to multiple files at once even with RotatingFileHandler?

I am using Django with logging config as follow:

LOGGING = {
    'version':1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{asctime} {process:d} {thread:d} {levelname} {name} {module} {funcName} {message}',
            'style': '{',
        }
    },
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': BASE_DIR+'/logs/django_logs.log',
            'backupCount': 14,
            'maxBytes': 52428800,
            'formatter': 'verbose'
        }
    },
    'loggers': {
        '': {
            'handlers': ['file'],
            'level': 'INFO'
        }
    },
}

I am running 16 processes of django, few daphne for websockets and few gunicorn processess for normal API calls based on django rest framework. But when I am looking at logs, multiple files are getting logged at once. For example, django_logs.1 .... django.logs.14 are getting logged into simultaneously. Do I have to add something else to log to one file at once and rotate it only when it's size exceed the specified size of log file?

For extra info, I am using python 3.6.8 and I initialize logger in each project file as follow:

import logging
logger = logging.getLogger(__name__)

Upvotes: 2

Views: 1981

Answers (1)

cr_alberto
cr_alberto

Reputation: 352

From docs.python.logging-cookbook:

Although logging is thread-safe, and logging to a single file from multiple threads in a single process is supported, logging to a single file from multiple processes is not supported, because there is no standard way to serialize access to a single file across multiple processes in Python. If you need to log to a single file from multiple processes, one way of doing this is to have all the processes log to a SocketHandler, and have a separate process which implements a socket server which reads from the socket and logs to file. (If you prefer, you can dedicate one thread in one of the existing processes to perform this function.)

A complete example of this approach is detailed also in the cookbook. See this section.

The docs also suggest a few alternatives to this approach:

Alternatively, you can use a Queue and a QueueHandler to send all logging events to one of the processes in your multi-process application

I've tested your logging configuration and it's working as expected when an unique process is logging to file handler. The file should rotate only when maxBytes is reached

Upvotes: 6

Related Questions