CODEWITHSUNDEEP
pythonpython-3.xamazon-web-servicesamazon-cloudwatchaws-glue
HouKaide
HouKaide

Reputation: 333

Missing Logs in AWS Glue Python

I have inherited a python script that I'm trying to log in Glue. Originally it had prints, but they were only sent once job finished, but it was not possible to see the status of the execution in running time.

I've changed the log system to the cloudwatch one, but apparently it doesn't send the logs in streaming way as the Spark one, according to this.

I decided to follow what they recommended and use watchtower for this purposes and I have a code like that:

def initialize_log() -> logging.Logger:
    logger = logging.getLogger(__name__)
    log_format = "[%(name)s] %(asctime)s %(levelname)-8s %(message)s"

    date_format = "%a, %d %b %Y %H:%M:%S %Z"
    log_stream = sys.stdout

    logging.basicConfig(level=logging.INFO, format=log_format, stream=log_stream, datefmt=date_format)
    logger.addHandler(watchtower.CloudWatchLogHandler(log_group='/aws-glue/python-job', stream_name='my_stream_name'))
    return logger

def log(logger, message):
    logger.info(message)
    logger.info(dict(foo="bar", details={}))

However, when I do:

logger = initialize_log()

log(logger, "Message")

I cannot find this into message in Cloudwatch /aws-glue/python-job or any directory.

I would like to ask if you know what I may be doing wrong.

Thank you in advance

Upvotes: 1

Views: 1883

Answers (1)

HouKaide
HouKaide

Reputation: 333

Solved with logging library:

import logging

def initialize_log() -> logging.Logger:
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)

    handler = logging.StreamHandler(sys.stdout)
    handler.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    return logger


def log(message: str):
    logger.info(message)

Upvotes: 2

Related Questions