CODEWITHSUNDEEP
djangocelery
KVISH
KVISH

Reputation: 13178

celery logging with django

I'm using celery with django. For a long time everything was working perfectly, but recently I was having issues with permissions on the log files for celery. Right now I have the following for my logging configuration:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '[%(levelname)s %(asctime)s %(thread)d] %(module)s:%(funcName)s - %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
        },
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'mail_admins_critical': {
            'level': 'CRITICAL',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.TimedRotatingFileHandler',
            'formatter': 'verbose',
            'filename': '/tmp/django.log',
            'when' : 'midnight',
            'interval' : 1
        },
        'celery': {
            'level': 'DEBUG',
            'class': 'logging.handlers.TimedRotatingFileHandler',
            'formatter': 'verbose',
            'filename': '/tmp/celery.log',
            'when' : 'midnight',
            'interval' : 1
        },
        'console': {
            'level': 'DEBUG',
            'formatter': 'verbose',
            'class': 'logging.StreamHandler'
        },
    },
    'loggers': {
        'main.logger': {
            'handlers': ['file', 'console', 'mail_admins_critical'],
            'level': 'INFO',
            'propagate': True,
        },
        'celery.logger': {
            'handlers': ['celery', 'console', 'mail_admins_critical'],
            'level': 'INFO',
            'propagate': True,
        },
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

For all celery files I put the following at the top: logger = get_task_logger('celery.logger')

I notice that the log file for celery has the user www-data and it doesn't have write permission when the log file is first created (celery.log). How can I change the permission so that by default it does? Only recently I started having this issue...

I read the following regarding this:

https://groups.google.com/forum/?fromgroups=#!topic/celery-users/wUHlE1piHu8

http://docs.celeryproject.org/en/latest/reference/celery.app.log.html

EDIT

When I change the line for logging to this:

logger = get_task_logger(__name__)

and put CELERYD_HIJACK_ROOT_LOGGER = False in my settings.py, it still hijacks my main logger (main.logger). Is there a way to stop that? It will then require permissions on that log file otherwise...

Upvotes: 2

Views: 4270

Answers (1)

asksol
asksol

Reputation: 19499

The generic init scripts in the celery git repo uses a dedicated directory for this, e.g.:

/var/log/celery/worker1.log
/var/log/celery/worker2.log
/var/log/celery/beat.log

Then you can just give www-data permissions to create and read files in this directory:

sudo chmod 755 /var/log/celery
sudo chown www-data:www-data /var/log/celery

You should not give write access to the /var/log top directory for non-privileged users, so using subdirectories is a good practice.

Upvotes: 1

Related Questions