CODEWITHSUNDEEP
pythonamazon-web-servicesloggingbotopython-logging
agiliq
agiliq

Reputation: 7748

Disable boto logging without modifying the boto files

I am using the Boto library to talk to AWS. I want to disable logging. (Or redirect to /dev/null or other file). I can't find an obvious way to do this. I tried this, but that doesn't seem to help:

import boto
boto.set_file_logger('boto', 'logs/boto.log')

This says it is possible, http://developer.amazonwebservices.com/connect/thread.jspa?messageID=52727&#52727 but as far as I know the documentation doesn't tell how.

Upvotes: 90

Views: 49956

Answers (9)

phoenix_sp
phoenix_sp

Reputation: 73

If you're looking to disable boto and/or other loggers because logs generated by your code are bloated with debug logs of other modules, you can avoid using root logger.

Instead of importing with logging.getLogger(), import with logging.getLogger('module_name').

Referred from a comment to an issue on official boto3 github repo.

Upvotes: 0

Renuá Meireles
Renuá Meireles

Reputation: 51

If you're dealing with boto3, it can be done by setting a general logging level in basicConfig. In the example below we try to check the existence of a file in a bucket on S3 service:

import boto3
import botocore
import logging

logging.basicConfig(..., level=logging.INFO) # ***** HERE IS THE SOLUTION *****

s3 = boto3.resource('s3', ...) # credetials
my_bucket = s3.Bucket('bucket_name')

logging.info('INFO TEST')
logging.error('ERROR TEST')
logging.warning('WARNING TEST')

try:
    my_bucket.Object('test/test.txt').load() # it will send some https requests and DEBUG them in logging
except botocore.exceptions.ClientError as e:
    if e.response['Error']['Code'] == "404":
        print('The object does not exist.')
else:
    print('The object does exist.')

Upvotes: 0

Yannick Schuchmann
Yannick Schuchmann

Reputation: 502

Beware this solution will disable also non-boto logs. See mchlfchr's answer instead.

For me none of the posted solutions worked unfortunately. Probably due to meanwhile changes in boto itself.

But sometimes a look into the manual does help..

import logging
import boto3
boto3.set_stream_logger('', logging.CRITICAL)

Upvotes: 2

Aleksandra
Aleksandra

Reputation: 66

There is another annoying property of the logging module in python: if you import logging again (for instance because you import a function from a file where logging is imported) AFTER setting the loglevels for your code, setting the levels might not have any effect. I don't know the exact details of why that is, but I only managed to set correct log levels for imported libraries after implementing something like below, and ALWAYS using a logger created by that function. The file where this logger creation function is defined, is the only file in my code base where logging is imported.

import logging

def get_logger_by_name(logger_name: str, log_filepath: str = "/training.log") -> logging.Logger:
    """
    Function that reloads logging module to store logs into a file and creates logger
    :param logger_name: Name of the logger that is returned
    :param log_filepath: filepath to log_file to store all logs
    :return: logger object
    """
    reload(
        logging
    )  # we need to import logging again to configure log file. https://stackoverflow.com/a/53553516/11758585
    logging.basicConfig(
            level=logging.DEBUG,
            format="%(asctime)s %(levelname)-8s %(name)-10s %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S",
            handlers=[logging.FileHandler(log_filepath), logging.StreamHandler()],
        )
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)

    # clean messy log output by setting log level to WARNING for unimportant loggers
    logging.getLogger("botocore").setLevel(logging.WARNING)
    logging.getLogger("boto3").setLevel(logging.WARNING)
    logging.getLogger("boto").setLevel(logging.WARNING)
    logging.getLogger("s3transfer").setLevel(logging.WARNING)
    return logger

Upvotes: 0

Asclepius
Asclepius

Reputation: 63322

This answer is for those who're using logging.config.dictConfig.

It is recommended to disable DEBUG and INFO messages from all external packages, not limited to botocore and boto3:

LOGGING_CONFIG = { # Add your preexisting logging config here.
    "loggers": { # Add your preexisting loggers here.
        "": {"level": "WARNING", "handlers": ["console"], "propagate": False},  # Root logger.
     }

Alternatively, to disable debug messages from botocore and boto3 but not from all external packages:

LOGGING_CONFIG = { # Add your preexisting config here too.
    "loggers": { # Add your preexisting loggers here too.
        "botocore": {"level": "WARNING", "handlers": ["console"], "propagate": False},
        "boto3": {"level": "WARNING", "handlers": ["console"], "propagate": False},
     }

Assuming your logging configuration dict is named LOGGING, run this next:

logging.config.dictConfig(LOGGING)

The above must be run before boto3 is imported, irrespective of whether it is imported directly or indirectly! It won't entirely work if it's run after boto3 is already imported. You can choose to replace "WARNING" above with "INFO" or "ERROR" or "CRITICAL".

Upvotes: 8

mchlfchr
mchlfchr

Reputation: 4278

This is the only solution, which works for me as of today (2020/01/31):

for name in ['boto', 'urllib3', 's3transfer', 'boto3', 'botocore', 'nose']:
    logging.getLogger(name).setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

The solution with 

boto3.set_stream_logger('', logging.CRITICAL)

was killing my whole non-boto logs. It manipulates the root logger of the standard logging from python.

Try it out for yourself:

import logging
import boto3
import sys
logger = logging.getLogger(__name__)
boto3.set_stream_logger('', logging.CRITICAL)
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                    format='%(asctime)s - %(levelname)s - %(message)s')


if __name__ == '__main__':
    s3_client = boto3.client('s3')
    response = s3_client.list_buckets()
    logger.info(f'bucket list: {response}')

Regardless of where the init of the logger happens, it won't bring up the output. Remove the line of the boto3.set_stream_logger('', logging.CRITICAL) and the non-boto3 logs will re-appear again! Consequently the only working solution is NOT to use the approach with boto3.set_stream_logger() and apply it as I suggested.

Upvotes: 22

lony
lony

Reputation: 7780

I move the boto3 answer from the comments (namely charneykaye and gene_wood) to a proper answer:

import logging

logger = logging.getLogger()
logger.addHandler(logging.StreamHandler()) # Writes to console
logger.setLevel(logging.DEBUG)
logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)
logging.getLogger('s3transfer').setLevel(logging.CRITICAL)
logging.getLogger('urllib3').setLevel(logging.CRITICAL)

import boto3

s3 = boto3.resource('s3')

for bucket in s3.buckets.all():
    print(bucket.name)

To get all the loggers follow the response from leobarcellos:

import logging
loggers_dict = logging.Logger.manager.loggerDict

Upvotes: 75

Doug Harris
Doug Harris

Reputation: 3379

Better yet, disable propagate for boto:

import boto
boto.set_file_logger('boto', 'logs/boto.log')
logging.getLogger('boto').propagate = False

Upvotes: 12

Vinay Sajip
Vinay Sajip

Reputation: 99375

You could try

import logging
logging.getLogger('boto').setLevel(logging.CRITICAL)

which will suppress all (other than CRITICAL) errors.

Boto uses logging configuration files (e.g. /etc/boto.cfg, ~/.boto) so see if you can configure it to your needs that way.

The set_file_logger call simply adds a user-defined file to the logging setup, so you can't use that to turn logging off.

Upvotes: 113

Related Questions