CODEWITHSUNDEEP
pythonloggingetlaws-glue
jfhfhf839
jfhfhf839

Reputation: 1739

AWS Glue Python ETL: logger messages appear in the error cloudwatch stream

I'm writing a Glue ETL, and I'm trying to log using Python default's logger.

The problem is that all the log messages I'm printing using the logger appear in the error stream of the job.

If I print directly to stdout (using print), I see the printed messages in the regular cloudwatch log stream.

I tried to redirect my logger to stdout, but I still got the same result: messages appear in error stream.

Does anyone know how I can use logger, and still see my messages in the log cloudwatch stream? (and not in the error cloudwatch stream)

This is the code sample I'm using to test:

import logging
import sys

MSG_FORMAT = '%(asctime)s %(levelname)s %(name)s: %(message)s'
DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=MSG_FORMAT, datefmt=DATETIME_FORMAT, stream=sys.stdout)
logger = logging.getLogger()

logger.setLevel(logging.INFO)

logger.info("Test log message. This appear on the job error cloudwatch stream")

print("This is a print. This appear on the job log cloudwatch stream")

Upvotes: 2

Views: 1891

Answers (2)

RobertS
RobertS

Reputation: 1

logging.basicConfig() does not do anything if the root logger already has handlers (see docs).

To fix this, pass force=True to logging.basicConfig().

Background

It seems to me like Glue jobs already have logging configured at application startup:

Running

import logging

root = logging.getLogger()
print(f"{root.level=}, {root.handlers=}")
for handler in root.handlers:
    print(f"{handler.level=}, {getattr(handler, 'stream', None)=}")

outputs

root.level=30, root.handlers=[<StreamHandler <stderr> (NOTSET)>]
handler.level=0, getattr(handler, 'stream', None)=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>

This means the root logger is at the default level "WARNING" and has one handler writing to stderr.

stderr logs are separately displayed in Cloudwatch as "Error logs".

Upvotes: 0

jfhfhf839
jfhfhf839

Reputation: 1739

I ended up adding

logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))

to my logger definition. That resolved the problem

Upvotes: 3

Related Questions