CODEWITHSUNDEEP
pythonpython-3.xlogging
fyrkov
fyrkov

Reputation: 2715

How can I decorate python logging output?

I use import logging module for logging inside the AWS lambda with python 3.7 runtime.

I would like to perform certain manipulations on log statements before they are flushed to stdout, e.g. wrap the message as json and add tracing data, so that they would be parseable by Kibana parser.

I don't want to write my own decorator for that because that won't work for underlying dependencies.

Ideally, it should be something like a configured callback for the logger so that it would do following work for me:

    log_statement = {}
    log_statement['message'] = 'this is the message'
    log_statement['X-B3-TraceId'] = "76b85f5e32ce7b46"
    log_statement['level'] = 'INFO'
    sys.stdout.write(json.dumps(log_statement) + '\n')

while having still logger.info('this is the message').

How can I do that?

Upvotes: 1

Views: 971

Answers (1)

fyrkov
fyrkov

Reputation: 2715

Answering my own question:

  1. I had to use LoggerAdapter that is quite a good fit for the purpose of pre-processing log statements:
import logging

class CustomAdapter(logging.LoggerAdapter):

    def process(self, msg, kwargs):
        log_statement = '{"X-B3-TraceId":"%s", "message":"%s"}' % (self.extra['X-B3-TraceId'], msg) + '\n'
        return log_statement, kwargs

See: https://docs.python.org/3/howto/logging-cookbook.html#using-loggeradapters-to-impart-contextual-information

  1. In general, the next step would be just plugging in the adapter like:
import logging
...
logging.basicConfig(format='%(message)s')
logger = logging.getLogger()
logger.setLevel(LOG_LEVEL)
custom_logger = CustomAdapter(logger, {'X-B3-TraceId': "test"})
...
custom_logger.info("test")

Note: I had to put format as a message only because I need to get the whole statement as a JSON string. Unfortunately, thus I lost some predefined log statement parts, e.g. aws_request_id. This is the limitation of LoggerAdapter#process as it handles only the message part. If anyone has a better approach here, pls suggest.

  1. It appears that AWS lambda python runtime somehow interferes with logging facility and changing the format like above did not work. So I had to do additionally this:
FORMAT = "%(message)s"
logger = logging.getLogger()
for h in logger.handlers:
    h.setFormatter(logging.Formatter(FORMAT))

See: https://gist.github.com/niranjv/fb95e716151642e8ca553b0e38dd152e

Upvotes: 2

Related Questions