CODEWITHSUNDEEP
pythonloggingmodule
noname7619
noname7619

Reputation: 3778

Python module structure and logging

I'm trying to log to the same file from the main flow and submodule:

/proxy
/proxy/proxy/network.py
/proxy/proxy.py

network.py:

import logging
print __name__
...

logging.getLogger(__name__).warning("test warn")

proxy.py:

import logging
print __name__
logging.basicConfig(filename="console", format="%(message)s", disable_existing_loggers=False)
logging.warning("test main")

Then I see logging in proxy.py works, while in network.py doesn't, obviously because their name's are:

proxy.network
__main__

respecitvely.

If I put them to the same folder their names would be like:

network
__main__

which also wouldn't work. Looks like I have to tell somehow that my project root is one level up relative to proxy.py (wherever it is), in this case they both would have name "proxy.*".

How to point the project root folder to interpreter? How to unify their module structure names, to enable logging(in either module structure)?

Upvotes: 0

Views: 369

Answers (2)

aluriak
aluriak

Reputation: 5867

EDIT: here is an access to the file name related to the whole package:

filepath = __file__ if __name__ == '__main__' else __name__

For the following structure, it gives:

mypack
├── __main__.py   # -> mypack.__main__
└── sub
    ├── __init__.py
    └── sub.py  # -> mypack.sub.sub

Here is a snippet:

def logger(name=None, root_name=PACKAGE_NAME):
    if name:
        return logging.getLogger(root_name + '.' + name)
    else:
        return logging.getLogger(root_name)

I usually define this function in a commons.py or utils.py, imported by all the modules of my package.

Taking advantage of the sublogger system, this allows a package to use the main logging of the package:

import utils
logger = utils.logger()
logger.info('module xxx started')

Or a sublogger dedicated to a particular concept:

import utils
logger = utils.logger('vm')
logger.info('module vm.xxx started')

The file path and the logging config are now decoupled and explicitely binded. Which can be automatize for each module:

import utils
logger = utils.logger(__file__)
logger.info('module ' + __file__ + ' started')

Because of the sublogger system, the logger 'pk.vm' will (by default) inherit for 'pk' configuration, that is to say using the same handlers, thus writing in the same logging file. However, it can be useful to define some particular behavior or handlers for any sublogger:

logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s',
        },
        'simple': {
            'format': '%(levelname)s %(message)s',
        },
    },
    'handlers': {
        'console':{
            'level':LOG_LEVEL,
            'class':'logging.StreamHandler',
            'formatter': 'simple',
        },
        'logfile': {
            'level': LOG_LEVEL,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': DIR_LOGS + LOGGER_NAME + '.log',
            'mode': 'w',
            'maxBytes': LOGFILE_MAX_SIZE,
            'formatter': 'verbose',
        },
        'logfile' + SUBLOGGER_SEPARATOR + SUBLOGGER_VM: {
            'level': LOG_LEVEL,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': DIR_LOGS + LOGGER_NAME + '.' + SUBLOGGER_VM + '.log',
            'mode': 'w',
            'maxBytes': LOGFILE_MAX_SIZE,
            'formatter': 'verbose',
        },
    },
    'loggers': {
        PACKAGE_NAME: {
            'handlers':['console', 'logfile'],
            'propagate': True,
            'level':LOG_LEVEL,
        },
        PACKAGE_NAME + SUBLOGGER_SEPARATOR + SUBLOGGER_VM: {
            'handlers':['logfile' + SUBLOGGER_SEPARATOR + SUBLOGGER_VM],
            'level':LOG_LEVEL,
        },
    }
})

Upvotes: 1

akash12300
akash12300

Reputation: 422

Simplest approach to save all logging info to same file is, do NOT provide a parameter in getLogger() [instead of what you are using now, getLogger(name)]

But keep in mind that if you are using some other module which use logging (outside your project, some third party maybe, like httplib) they might end up saving data in your log file with this approach

Upvotes: 1

Related Questions