CODEWITHSUNDEEP
pythonloggingstdoutstderrobservability
Pavel Durov
Pavel Durov

Reputation: 1297

Python default and multi level logging

From what I read and understood, Python logging module by default logs to stderr.

If I run this python code:

import logging


logging.info('test')
logging.warning('test')
logging.error('test')
logging.debug('test')

as

python main.py 1> stdout.txt 2> stderr.txt

I get my logs in stderr.txt and nothing in stdout.txt - my logs are redirected to stderr.

This default behaviour is problematic when logs are streamed to logging aggregation services such as datadog or papertrail. Since its streamed to stderr, the logs are marked as errors when in reality they are not.

So I tried to create multiple log handlers as follows:

import logging
import sys


stdoutHandler = logging.StreamHandler(stream=sys.stdout)
stderrHandler = logging.StreamHandler(stream=sys.stderr)

logging.basicConfig(level=logging.DEBUG, handlers=[stdoutHandler, stderrHandler])

stdoutHandler.setLevel(logging.DEBUG)
stderrHandler.setLevel(logging.ERROR)


logging.info('test')
logging.warning('test')
logging.error('test')
logging.debug('test')

When I run this code, I get errors in sterr.txt but also all the logs in stdout.txt - I ended up having log duplication error logs appear in both the stderr and stdout streams.

Is there a better way to handle the differentiation of error logs from the rest in Python?

I tried loguru package as well, also no luck in stream separation... Thanks in advance

Upvotes: 0

Views: 377

Answers (2)

Pavel Durov
Pavel Durov

Reputation: 1297

Following the approach described in the official documentation: https://docs.python.org/3.10/howto/logging-cookbook.html#custom-handling-of-levels

Here's my code with stream segregation:

import sys
import logging



class StdoutFilter(logging.Filter):
    def filter(self, record):
        return record.levelno >= ROOT_LEVEL and record.levelno < logging.ERROR


class StderrFilter(logging.Filter):
    def filter(self, record):
        return record.levelno >= logging.ERROR



stdoutHandler = logging.StreamHandler(stream=sys.stdout)
stdoutHandler.addFilter(StdoutFilter())

stderrHandler = logging.StreamHandler(stream=sys.stderr)
stderrHandler.addFilter(StderrFilter())


root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)

root_logger.addHandler(stderrHandler)
root_logger.addHandler(stdoutHandler)

Upvotes: 0

Vinay Sajip
Vinay Sajip

Reputation: 99415

You can probably benefit from the approach described here in the official documentation.

Upvotes: 1

Related Questions