Django + Loggly: app logs not saved

Question

I'm using Loggly in order to have a centralized logs aggregator for my app running on AWS (Elastic beanstalk). However I'm not able to save my application logs using the Python logging library and the django logging configuration. In my Loggly control panel I can see a lot of logs coming from the underlying OS and software stack of my EC2 instance, but spefic logs from my app are not displayed and I don't understand why. I configured Loggly by configuring RSYSLOG on my EC2 instance (using the automated python script provided by Loggly itself), then I defined the following in my django settings:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            'format': '[%(asctime)s] [%(levelname)s] [%(name)s:%(lineno)s] %(message)s',
            'datefmt': '%d/%b/%Y %H:%M:%S'
        },
        'loggly': {
            'format': 'loggly: %(message)s',
        },
    },
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler',
            'formatter': 'standard',
        },
        'syslog': {
            'level': 'INFO',
            'class': 'logging.handlers.SysLogHandler',
            'facility': 'local5',
            'formatter': 'loggly',
        },
        'cygora': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/tmp/cygora.log',
            'maxBytes': 1024 * 1024 * 5, # 5 mb
            'backupCount': 10,
            'formatter': 'standard',
        },
        'django': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/tmp/django.log',
            'maxBytes': 1024 * 1024 * 5, # 5 mb
            'backupCount': 10,
            'formatter': 'standard',
        },
        'celery': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/tmp/celery.log',
            'maxBytes': 1024 * 1024 * 5, # 5 mb
            'backupCount': 10,
            'formatter': 'standard',
        },
    },
    'loggers': {
        'loggly': {
            'handlers': ['syslog'],
            'propagate': True,
            'format': 'loggly: %(message)s',
            'level': 'DEBUG',
        },
        'django': {
            'handlers': ['syslog', 'django'],
            'level': 'WARNING',
            'propagate': True,
        },
        'django.db.backends': {
            'handlers': ['syslog', 'django'],
            'level': 'INFO',
            'propagate': True,
        },
        'django.request': {
            'handlers': ['syslog', 'mail', 'django'],
            'level': 'INFO',
            'propagate': True,
        },
        'celery': {
            'handlers': ['syslog', 'mail', 'celery'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'com.cygora': {
            'handlers': ['syslog', 'cygora'],
            'level': 'INFO',
            'propagate': True,
        },
    }
}

In my classes I use the "standard" approach of having a module-level logger:

import logging
logger = logging.getLogger(__name__)
logger.info('This message is not displayed on Loggly!! :(')

but it doesn't work, neither using:

import logging
logger = logging.getLogger('loggly')
logger.info('This message is not displayed on Loggly!! :(')

Any idea? (is there someone using Django + Loggly with RSYSLOG)

Answer 1

The problem is either in the local rsyslog service receiving the logs or in sending them. Your LOGGING setting is solid, but since you are taking control of everything (like the Django loggers) you should set 'disable_existing_loggers': True. (Minor point: you can drop 'format' from the loggly loggers; the syslog handler will do that with its 'formatter'.)

Following the Loggly Django example, there are two steps left.

1. Setup Syslog with these instructions. You've done this and it sounds like it's working.

2. Make sure the service is listening by uncommenting these lines in rsyslog.conf:

# provides UDP syslog reception $ModLoad imudp $UDPServerRun 514

Verify rsyslog's config with rsyslog -N1 and restart the service. Rsyslog troubleshooting mentions looking at the rsyslog log and running the service interactively; hopefully you don't have to go to those depths. Look at Loggly's gotchas section first.

Start a Django shell and test it (which you're already doing—nice work!).

> manage.py shell
import logging
logger = logging.getLogger('loggly')
logger.info('This message is not displayed on Loggly!! :(')

Also, you do not have a root logger. So it's quite likely that logging.getLogger(__name__) won't be caught and handled. That's a general note; your efforts are thorough and not limited by this.

Answer 2

Googled around and saw your post on loggly's support. Did you see their reply and did it help you?

http://community.loggly.com/customer/portal/questions/5898190-django-loggly-app-logs-not-saved