CODEWITHSUNDEEP
pythonlogging
mmvdv
mmvdv

Reputation: 526

Use python logging for memory usage statistics

I work on a project which uses python's logging module for displaying messages. I have my own project, which uses a module from a common repository, so I don't want to change any of the logging statements in that part of the code.

However, memory usage appears to be quite an issue in my program, so I'd like to log memory usage in each log statement, alongside the time and the message, like this:

YYYY-MM-DD HH:MM:SS,mmm NAME LEVEL MemTotal:#M,Swap:#M MESSAGE

Is there a straightforward way to do this using the logging module?

I think the way to go is by using a Filter to add contextual information (cf. http://docs.python.org/howto/logging-cookbook.html#filters-contextual), but there does not seem to be a way to add this Filter to all levels and all instances of loggers at once. The documentation suggests adding the filter to the Handler instead of to the Logger, because otherwise it won't use the filter in external modules. Doing this in the most obvious way (creating a Handler, adding the Filter, and then attaching the Handler to the root Logger) however gave me unexpected behavior. I either got no output at all, and no error messages either, or (when using basicConfig first) I got the right behavior except that it gave error messages too. I suspect in the last case, I actually get two handlers, one of them working correctly, the other erroneously.

So far, I've come up with the following solution, which I don't think is quite elegant (thanks though to https://stackoverflow.com/a/938800/819110). The ugly part is that I have to extract the Handler from the Logger by hand, and add the Filter to it. I can't seem to add a Filter to the config file, which would be much more convenient. Nevertheless, this approach seems to work (on Linux), even though I suspect there should be a more straightforward way to do this.

import logging
import external_module

class MemuseFilter(logging.Filter):
    def filter(self, record):
        """ This function overrides logging.Filter, adds memuse as a field
        """ 
        record.memuse = self.str_mem()
        return True

    # Following code from https://stackoverflow.com/a/938800/819110: 
    _proc_status = '/proc/%d/status' % os.getpid()
    _scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
              'KB': 1024.0, 'MB': 1024.0*1024.0}

    def _VmB(self,VmKey):
        """Private.
        """
        # get pseudo file  /proc/<pid>/status
        try:
            t = open(self._proc_status)
            v = t.read()
            t.close()
        except:
            return 0.0  # non-Linux?
        # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'
        i = v.index(VmKey)
        v = v[i:].split(None, 3)  # whitespace
        if len(v) < 3:
            return 0.0  # invalid format?
        # convert Vm value to bytes
        return float(v[1]) * self._scale[v[2]]

    def memory(self,since=0.0):
        """Return memory usage in bytes.
        """
        return self._VmB('VmSize:') - since

    def swapsize(self,since=0.0):
        """Return swap size in bytes.
        """
        return self._VmB('VmSwap:') - since

    def byte_to_mb(self,byte):
        """return size in MB (being lazy)
        """
        return byte/(1024*1024)

    def str_mem(self):
        """Return a string with the total memuse and swap size in MB
        """
        return "MemTotal:%.0fM,Swap:%.0fM"%(\
            self.byte_to_mb(self.memory()),self.byte_to_mb(self.swapsize()) )

if __name__ == '_main__':
    logging.config.fileConfig('logging.conf') # Get basic config
    log = logging.getLogger('')               # Get root logger
    f = MemuseFilter()                        # Create filter
    log.handlers[0].addFilter(f)         # The ugly part:adding filter to handler
    log.warning("Foo")
    function_from_module_using_logging()

Where the external_module reads something like this:

log = logging.getLogger(__name__)
def function_from_module_using_logging():
    log.warning("Bar")

and the logging.conf is like this:

[loggers]
keys=root

[handlers]
keys=fileHandler

[formatters]
keys=memuseFormatter

[logger_root]
level=DEBUG
handlers=fileHandler

[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=memuseFormatter
args=('some.log','w')

[formatter_memuseFormatter]
format=%(asctime)-15s %(name)-5s %(levelname)-8s %(memuse)-22s %(message)s
datefmt=

Better solutions would be most welcome!

EDIT: Referenced the wrong SO Question.

Upvotes: 4

Views: 4163

Answers (2)

Vinay Sajip
Vinay Sajip

Reputation: 99335

If you're using Python 2.7 or better, you can use dictConfig(), which does support filters in the configuration. If you're using an earlier version of Python, you can use the logutils project to give you the dictConfig() functionality. If you don't want an external dependency on logutils, you can copy the dictConfig() stuff from there into your own code (as Django did/does).

Upvotes: 1

Aaron Digulla
Aaron Digulla

Reputation: 328566

First, let's clean up a misconception: A logger isn't the right place to add this information - loggers are little more than look up keys for log levels in a dict. There is no point to add memory usage to each logger because the formatter isn't going to look there.

The correct solution is to write your own formatter (as you did) and implement a new variable. There is no better way to do this. For example, you don't want to get memory information for logger calls that are never written to the log file.

Upvotes: 1

Related Questions