CODEWITHSUNDEEP
python-3.xloggingaws-lambda
Guillaume
Guillaume

Reputation: 3051

Disable logging on AWS Lambda Python 3.9

I have some code that is using logger package to log to a file on a Lambda function. The problem is that it also sends everything to CloudWatch, and since there are a lot of logs, is very expensive.

When I was using Python 3.7, this was working and logging only to the file:

import os
import sys
import logging

LOG_LEVEL = os.getenv('LOG_LEVEL', 'INFO')

root_logger = logging.getLogger()
root_logger.disabled = True

# create custom logger
logger = logging.getLogger('my_logger')
logger.removeHandler(sys.stdout)
logger.setLevel(logging.getLevelName(LOG_LEVEL))

for handler in logger.handlers:
    logger.removeHandler(handler)

handler = logging.FileHandler('file.log', encoding='utf-8')
handler.setLevel(logging.getLevelName(LOG_LEVEL))
logger.addHandler(handler)

sys.stdout = open(os.devnull, 'w')
run_code_that_logs_stuff()

But after upgrading to Python 3.9, the logs started to show in CloudWatch again.

I have tried changing the last 2 lines with:

with open(os.devnull, 'w') as f, contextlib.redirect_stdout(f):
    run_code_that_logs_stuff()

but same result:

START RequestId: 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Version: $LATEST


2022-05-29T05:46:50.949+02:00   [INFO] 2022-05-29T03:46:50.949Z 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Printing stuff I don't want to appear

Upvotes: 0

Views: 1910

Answers (1)

Parsifal
Parsifal

Reputation: 4486

The easiest way to prevent Lambda from writing to CloudWatch Logs is to remove the IAM policy that allows it to do so.

If you look at your Lambda's configuration, you'll see an execution role. If you look at that role's configuration, you'll see that it references the managed policy AWSLambdaBasicExecutionRole (or, if it's running inside a VPC, AWSLambdaVPCAccessExecutionRole). If you look at those managed policies, you'll see that they grant permissions logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents.

Replace those managed policies with your own managed policies that grant everything except those three permissions.

Or, alternatively, create a managed policy that explicitly denies those three permissions (and nothing else), and attach it to the Lambda's execution role.

Update:

I noted in the comments that you can remove all handlers from the logging module at the start of your Lambda code. However, I think that's an incredibly bad idea, as it means that you will have no visibility into your function if things go wrong.

A much better approach would be to leverage the capabilities of the Python logging framework, and log your output at different levels. At the top of your function, use this code to set the level of the logger:

LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(os.environ.get('LOG_LEVEL', 'WARNING'))

Then, in your code, use the appropriate level when logging:

LOGGER.debug("this message won't normally appear")
LOGGER.warning("this message will", exc_info=True)

This should trim your log messages considerably, while still giving you the ability to see what's happening in your function when things go wrong.

Docs: https://docs.python.org/3/library/logging.html

Update 2:

If you're determined to disable logging entirely (imo a bad idea), then I recommend doing it within the logging framework, rather than hacking streams:

import logging

logging.root.handlers = [logging.NullHandler()]


def lambda_handler(event, context):
    logging.warn("this message won't appear")

Upvotes: 2

Related Questions