CODEWITHSUNDEEP
jsonpython-3.x
user3501221
user3501221

Reputation: 21

JSON-logging: Implementing a custom log formatter to a single handler

Python: 3.7 json_logging: 1.3.0

I have a need to implement JSON-format logger for a Python application (for integration with Logstash). Currently, I have two different handlers implemented (one logging.StreamHandler for logging to console in a more "human" readable format, one logging.handlers.RotatingFileHandler).

I would like to retain the possibility of logging with these different handlers, as having JSON formatted output on console is not very friendly. I am using json_logging library for formatting the JSON logging part.

The issue I am having is trying to setup a custom JSON formatter. The default one has a lot of redundant information which I would like to remove (example, because I have custom logging function which create a log message from few input parameters, JSON formatted parameters "module" and "line_no" will always carry the same information (of my custom log function) and not the actual line number and module).

Logging setup is read from a configuration yaml file:

---
version: 1
disable_existing_loggers: False
formatters:
    console_formatter:
        format: "%(asctime)s %(process)d %(thread)d %(levelname)s %(message)s"
        datefmt: "%Y-%m-%dT%H:%M:%S%z"
    file_formatter:
        format: "%(asctime)s %(process)d %(thread)d %(levelname)s %(message)s"
        datefmt: "%Y-%m-%dT%H:%M:%S"
        class: json_logging.JSONLogFormatter
         
handlers:
    console:
        class: logging.StreamHandler
        level: DEBUG
        formatter: console_formatter
        stream: ext://sys.stdout
    
    # Default log file gets written into a file inside the project folder
    debug_file_handler:
        class: logging.handlers.RotatingFileHandler
        level: DEBUG
        formatter: file_formatter
        filename:  /var/log/runtime.log
        maxBytes: 10485760 # 10MB
        backupCount: 30
        encoding: utf8

loggers:
    my_module:
        level: DEBUG
        handlers: [console]
        propagate: no

root:
    level: DEBUG
    handlers: [console, debug_file_handler]

Snippet of the setup_logging function which reads the parameter file and sets up the logging:

        if path is None:
            logging.warning("Log configuration file NOT found: " + ','.join(value, default_path, package_path))
            logging.basicConfig(level=default_level, format=format)
        else:
            logging.debug("Log configuration file found: " + path)
            with open(path, 'rt') as f:
                config = yaml.safe_load(f.read())
            logging.config.dictConfig(config)

Using the class: json_logging.JSONLogFormatter in one of the handlers I am able to log in JSON format in only file_formatter and retain the non-JSON structure in console.

The issue comes into picture when I try to setup a custom JSON formatter. I am using the example provided in library's github: https://github.com/bobbui/json-logging-python/blob/master/example/custom_log_format.py

There, the custom class is provided for the init call to setup function json_logging.init_non_web(custom_formatter=CustomJSONLog, enable_json=True)

The problem is, that this function configures all handlers to use the same format. Which also changes the console formatter output to JSON.

I have tried to use getLogger and then setting the formatter to only the fileHandler, however that still changes all formatters:

            log = logging.getLogger()
            for hdlr in log.handlers[:]:  # remove the existing file handlers
                if isinstance(hdlr, logging.FileHandler):
                    hdlr.setFormatter(json_logging.init_non_web(custom_formatter=CustomJSONLog, enable_json=True))

Could someone help me in finding the way to use a custom JSON formatter for only a single handler?

Upvotes: 2

Views: 4569

Answers (1)

sam2426679
sam2426679

Reputation: 3847

It seems like the problem you're having is due to json_logging not "playing nicely" with all the different ways that the stdlib logging can be configured.

This answer describes how to configure json-formatted logs using only the standard library.

After you do that, you can simply configure handler-specific formatters as you've already attempted to do in your question.

Upvotes: 3

Related Questions