CODEWITHSUNDEEP
pythonlogging
slayedbylucifer
slayedbylucifer

Reputation: 23502

Python logging module prints unnecessary messages

Please consider this dummy code.

$ cat dummy.py 
import logging
import time

from boto3.session import Session

# Logging Configuration
fmt = '%(asctime)s [%(levelname)s] [%(module)s] - %(message)s'
logging.basicConfig(level='INFO', format=fmt, datefmt='%m/%d/%Y %I:%M:%S')
logger = logging.getLogger()

def main():
    session = Session(region_name='us-west-2')
    client = session.client('ec2')
    response = client.describe_instances(InstanceIds=['i-11111111111111111'])

    logger.info('The instnace size is: %s', response[
                'Reservations'][0]['Instances'][0]['InstanceType'])

if __name__ == '__main__':
    main()

Output:

$ python3 dummy.py 
03/03/2017 08:47:00 [INFO] [credentials] - Found credentials in shared credentials file: ~/.aws/credentials
03/03/2017 08:47:01 [INFO] [connectionpool] - Starting new HTTPS connection (1): ec2.us-west-2.amazonaws.com
03/03/2017 08:47:02 [INFO] [dummy] - The instnace size is: t2.micro

Question: How to avoid below lines from being printed? 

03/03/2017 08:47:00 [INFO] [credentials] - Found credentials in shared credentials file: ~/.aws/credentials
03/03/2017 08:47:01 [INFO] [connectionpool] - Starting new HTTPS connection (1): ec2.us-west-2.amazonaws.com

If I change logging.basicConfig(level='INFO',... to logging.basicConfig(level='WARNING',... then Those messages are not printed, but then all my messages get logged with WARNING severity.

I just want the logging module to print the messages that I explicitly write using logger.info .... and nothing else. Hence I need any pointers on how to avoid the unnecessary messages from being printed.

Upvotes: 5

Views: 2783

Answers (2)

Aubrey Lavigne
Aubrey Lavigne

Reputation: 652

boto3 defines a function set_stream_logger that overrides the logging level for a specific boto3 namespace.

Prototype:

boto3.set_stream_logger(name='boto3', level=logging.DEBUG, format_string=None)

Example:

import boto3
boto3.set_stream_logger(name='botocore.credentials', level=logging.WARNING)
boto3.set_stream_logger(name='urllib3.connectionpool', level=logging.WARNING)

botocore/credentials.py loads the logger with this line:

logger = logging.getLogger(__name__)

In this file __name__ resolves to botocore.credentials, so that will be the logger name when calling set_stream_logger.

Note that the Starting new HTTPS connection message comes from urllib3, which isn't part of boto3. The boto3.set_stream_logger function still works because it acts on the logging module.

Upvotes: 4

slayedbylucifer
slayedbylucifer

Reputation: 23502

Solution:

import logging
import time

from boto3.session import Session

# Logging Configuration
fmt = '%(asctime)s [%(levelname)s] [%(module)s] - %(message)s'
logging.basicConfig(format=fmt, datefmt='%m/%d/%Y %I:%M:%S')
logger = logging.getLogger('LUCIFER')
logger.setLevel(logging.INFO)


def main():
    COUNTER = 3
    session = Session(region_name='us-west-2')
    client = session.client('ec2')
    response = client.describe_instances(InstanceIds=['i-0a912622af142b510'])

    logger.info('The instnace size is: %s', response[
                'Reservations'][0]['Instances'][0]['InstanceType'])

if __name__ == '__main__':
    main()

Output:

$ python3 dummy.py 
03/03/2017 10:30:15 [INFO] [dummy] - The instnace size is: t2.micro

Explanation: Earlier, I set the INFO level on root logger. Hence, all other loggers who do not have a level set, get this level propagated and start logging. In the solution, I am specifically enabling this level on a logger LUCIFER.

Reference: from: https://docs.python.org/3/howto/logging.html

Child loggers propagate messages up to the handlers associated with their ancestor loggers. Because of this, it is unnecessary to define and configure handlers for all the loggers an application uses. It is sufficient to configure handlers for a top-level logger and create child loggers as needed. (You can, however, turn off propagation by setting the propagate attribute of a logger to False.)

AND

In addition to any handlers directly associated with a logger, all handlers associated with all ancestors of the logger are called to dispatch the message (unless the propagate flag for a logger is set to a false value, at which point the passing to ancestor handlers stops).

Upvotes: 1

Related Questions