CODEWITHSUNDEEP
amazon-ecsamazon-cloudwatchlogsaws-fargate
Aayush Karki
Aayush Karki

Reputation: 841

How to send logs to Cloudwatch from a background process running in AWS Fargate containers?

I'm using Fargate. My container is running two processes. Celery worker in the background and Django in the foreground. The foreground process emits logs to stdout, hence AWS takes care of sending Django logs to concerned Cloudwatch Log Group and Stream.

Since its running in the background, how do send the celery worker's logs to (a different Log Stream within same Log Group) Cloudwatch?

Upvotes: 2

Views: 2724

Answers (3)

claytond
claytond

Reputation: 1099

Like @OK999 said, Celery is designed to swallow logs whether its on Fargate or not. We ended up using a Django LOGGING config like:

LOGGING = {
    'version': 1,
    # This only "disables" but the loggers don't propagate
    # 'disable_existing_loggers': False,
    ...
    'handlers': {
        'console': {
            'level': env.str('LOGGING_LEVEL'),
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        ...
        # celery won't route logs to console without this
        'celery': {
            # filtered at the handler
            'level': logging.DEBUG,
            'handlers': ['console'],
        },
        ...

We had to make this change long before transitioning to Fargate.

Upvotes: 0

Igor Borisov
Igor Borisov

Reputation: 94

If there's no way to move the second process to the separate container and log it as usual, you may install awslogs package to the container and set it up to read background process' log files and send content to CloudWatch. But I'd not recommend such approach.

Upvotes: 2

OK999
OK999

Reputation: 1399

Again this is not necessarily a Fargate based issue or question. For logging in celery check this out http://docs.celeryproject.org/en/latest/userguide/tasks.html#logging. 

The worker won’t update the redirection if you create a logger instance somewhere in your task or task module.

If you want to redirect sys.stdout and sys.stderr to a custom logger you have to enable this manually, for example:

import sys

logger = get_task_logger(__name__)

@app.task(bind=True)
def add(self, x, y):
    old_outs = sys.stdout, sys.stderr
    rlevel = self.app.conf.worker_redirect_stdouts_level
    try:
        self.app.log.redirect_stdouts_to_logger(logger, rlevel)
        print('Adding {0} + {1}'.format(x, y))
        return x + y
    finally:
        sys.stdout, sys.stderr = old_outs

And for logging with Fargate, i would use the awslogs driver. Below is how you configure as documented here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html

If using the console:

enter image description here

Or this if in cloudformation template:

enter image description here

Upvotes: 1

Related Questions