Watchtower CloudWatch Integration in Django Logs: No Log Streams Created

I have a Django project with Sentry for error tracking, and logs are stored locally on the server. I want to forward these logs to CloudWatch Logs. I am using Watchtower for this integration.

Here's my settings.py:

############# LOGGING ###########
from boto3 import client
import watchtower

logger_boto3_client = client(
    "logs",
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    region_name=AWS_S3_REGION_NAME,
)

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "verbose": {
            "format": "[{levelname}] [{asctime}] [{pathname}] [{lineno}] [{message}]",
            "style": "{",
        },
    },
    "filters": {
        "debug_filter": {
            "()": "server_backend.log_settings.DebugFilter",
        },
        "info_filter": {
            "()": "server_backend.log_settings.InfoFilter",
        },
        "warning_filter": {
            "()": "server_backend.log_settings.WarningFilter",
        },
        "error_filter": {
            "()": "server_backend.log_settings.ErrorFilter",
        },
    },
    "handlers": {
        "debug_file": {
            "level": "DEBUG",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/debug.log",
            "formatter": "verbose",
            "maxBytes": 52428800,
            "backupCount": 10,
            "filters": ["debug_filter"],
        },
        "info_file": {
            "level": "INFO",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/info.log",
            "formatter": "verbose",
            "maxBytes": 52428800,
            "backupCount": 10,
            "filters": ["info_filter"],
        },
        "warning_file": {
            "level": "WARNING",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/warning.log",
            "formatter": "verbose",
            "maxBytes": 52428800,
            "backupCount": 10,
            "filters": ["warning_filter"],
        },
        "error_file": {
            "level": "ERROR",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/error.log",
            "maxBytes": 52428800,
            "backupCount": 10,
            "formatter": "verbose",
        },
        "watchtower": {
            "level": "DEBUG",
            "class": "watchtower.CloudWatchLogHandler",
            "boto3_client": logger_boto3_client,
            "log_group": "server_backend",
            "stream_name": "{strftime:%Y-%m-%d}",
            "formatter": "verbose",
            "create_log_group": True,  
            "create_log_stream": True, 
        },
    },
    "loggers": {             
        "server_backend": {
            "handlers": ["debug_file", "info_file", "warning_file", "error_file", "watchtower"],
            "level": "DEBUG",
            "propagate": True,
        },
        "django.request": {
            "handlers": ["debug_file", "info_file", "warning_file", "error_file"],
            "level": "DEBUG",
            "propagate": True,
        },
    },
}

When I run the following code in the Django shell, it doesn't work and throws the following warning:

import logging

logger = logging.getLogger("server_backend")
logger.info("This message should be logged without issues.")
# Output:
# .../env/lib/python3.10/site-packages/watchtower/__init__.py:437: WatchtowerWarning: Received message after logging system shutdown
#   warnings.warn("Received message after logging system shutdown", WatchtowerWarning)
# INFO:server_backend:This message should be logged without issues.

This creates a log group but no log streams are present. I have verified all permissions (full access to CloudWatch) and credentials (they work for other services such as S3).

However, when I add the following code to the settings:

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("server_backend")
logger.addHandler(watchtower.CloudWatchLogHandler(log_group="server_backend", stream_name="server_backend", create_log_group=True, create_log_stream=True, boto3_client=logger_boto3_client))
logger.info("Hi")
logger.info(dict(foo="bar", details={}))

I find the 'Hi' and the dict in my stream but no other logs and it just keeps on throwing the same warning as earlier on any cutom error i want to raise.

What could be causing the issue with the logging setup in my settings.py? How can I ensure logs are forwarded to CloudWatch without these warnings?

Upvotes: 1

Views: 136

Answers (0)

Related Questions