CODEWITHSUNDEEP
pythonjsonpython-3.xlogging
Mingo
Mingo

Reputation: 936

Extract the extra fields in logging call in log formatter

I can add additional fields to my log message like so:

logging.info("My log Message", extra={"someContext":1, "someOtherContext":2})

which is nice, but unclear how to extract all the extra fields in my log formatter:

def format(self, record):
  record_dict = record.__dict__.copy()
  print(record_dict)

In the above, I can see all my extra fields in the output dict, but they are flattened into a dict with lots of other junk I don't want, for instance:

{'name': 'root', 'msg': 'My log Message', 'args': (), 'levelname': 'INFO', 'levelno': 20, 'pathname': '.\\handler.py', 'filename': 'handler.py', 'module': 'handler', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 27, 'funcName': 'getPlan', 'created': 1575461352.0664868, 'msecs': 66.48683547973633, 'relativeCreated': 1253.0038356781006, 'thread': 15096, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 23740, 'someContext': 1, 'someOtherContext':2}

Is there any way of getting all my extra keys without having to know them all upfront?

I am writing a json formatter and want to create a dict, for instance:

justMyExtra = ?????
to_log = {
"message" record_dict["message"], 
**justMyExtra
}

Upvotes: 8

Views: 10665

Answers (4)

Chris
Chris

Reputation: 34551

One could use a similar approach to this answer, where a dictionary key, namely extra_info, is used to pass all the extra data to it. Then, inside the custom formatter, one could look for that key, in order to extract the data and add them to the log record.

Working Example

import logging, json


class CustomJSONFormatter(logging.Formatter):
    def __init__(self, fmt):
        logging.Formatter.__init__(self, fmt)

    def format(self, record):
        logging.Formatter.format(self, record)
        return json.dumps(get_log(record), indent=2)


def get_log(record):
    d = {
        "time": record.asctime,
        "level": record.levelname,
        "message": record.message,
    }

    if hasattr(record, "extra_info"):
        d["foo"] = record.extra_info["foo"]

    return d


logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setFormatter(CustomJSONFormatter('%(asctime)s'))
logger.addHandler(handler)
logger.info("test", extra={"extra_info": {"foo": "bar"}})

Output:

{
  "time": "2024-10-27 21:45:15,180",
  "level": "INFO",
  "message": "test",
  "foo": "bar"
}

Upvotes: 0

John Vandivier
John Vandivier

Reputation: 2426

Two approaches similar to those from @james-hendricks, but a little less fragile:

  1. Use the existing args dict on the LogRecord instead of coming up with your own dict.
  2. Create a dummy LogRecord and inspect its keys to get a non-fragile str list of LogRecord special keys:
  dummy = logging.LogRecord('dummy', 0, 'dummy', 0, None, None, None, None, None)
  reserved_keys = dummy.__dict__.keys()

Upvotes: 0

blhsing
blhsing

Reputation: 107015

If you read the source code of the logging.Logger.makeRecord method, which returns a LogRecord object with the given logging information, you'll find that it merges the extra dict with the __dict__ attribute of the returing LogRecord object, so you cannot retrieve the original extra dict afterwards in the formatter.

Instead, you can patch the logging.Logger.makeRecord method with a wrapper function that stores the given extra dict as the _extra attribute of the returning LogRecord object:

def make_record_with_extra(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None):
    record = original_makeRecord(self, name, level, fn, lno, msg, args, exc_info, func, extra, sinfo)
    record._extra = extra
    return record

original_makeRecord = logging.Logger.makeRecord
logging.Logger.makeRecord = make_record_with_extra

so that:

class myFormatter(logging.Formatter):
    def format(self, record):
        print('Got extra:', record._extra) # or do whatever you want with _extra
        return super().format(record)

logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.setFormatter(myFormatter('%(name)s - %(levelname)s - %(message)s - %(foo)s'))
logger.addHandler(handler)
logger.warning('test', extra={'foo': 'bar'})

outputs:

Got extra: {'foo': 'bar'}
__main__ - WARNING - test - bar

Demo: https://repl.it/@blhsing/WorthyTotalLivedistro

Upvotes: 4

james hendricks
james hendricks

Reputation: 134

Two approaches come to mind:

  1. Dump all of the extra fields into a dictionary within your main dictionary. Call the key "additionalContext" and get all the extra entries.
  2. Create a copy of the original dictionary and delete all of your known keys: 'name','msg','args', etc. until you only have justYourExtra

Upvotes: 2

Related Questions