CODEWITHSUNDEEP
pythoncode-coverageperformance-testing
guettli
guettli

Reputation: 27826

How can I measure the coverage (in production system)?

I would like to measure the coverage of my Python code which gets executed in the production system.

I want an answer to this question:

Which lines get executed often (hot spots) and which lines are never used (dead code)?

Of course this must not slow down my production site.

I am not talking about measuring the coverage of tests.

Upvotes: 7

Views: 1115

Answers (3)

Doruk Eren Aktaş
Doruk Eren Aktaş

Reputation: 2337

If i understand it right you want to learn which parts of your application is used most often by users.

TL;DR;

Use one of the metrics frameworks for python if you do not want to do it by hand. Some of them are above:

It is usually done by function level and it actually depends on application;

  • If it is a desktop app with internet access:

    You can create a simple db and collect how many times your functions are called. For accomplish it you can write a simple function and call it inside every function that you want to track. After that you can define an asynchronous task to upload your data to internet.

  • If it is a web application:

    You can track which functions are called from js (mostly preferred for user behaviour tracking) or from web api. It is a good practice to start from outer to go inner. First detect which end points are frequently called (If you are using a proxy like nginx you can analyze server logs to gather information. It is the easiest and cleanest way). After that insert a logger to every other function that you want to track and simply analyze your logs for every week or month.

But you want to analyze your production code line by line (it is a very bad idea) you can start your application with python profilers. Python has one already: cProfile.

Upvotes: 0

Ronald
Ronald

Reputation: 1039

I assume you are not talking about test suite code coverage which the other answer is referring to. That is a job for CI indeed.

If you want to know which code paths are hit often in your production system, then you're going to have to do some instrumentation / profiling. This will have a cost. You cannot add measurements for free. You can do it cheaply though and typically you would only run it for short amounts of time, long enough until you have your data.

Python has cProfile to do full profiling, measuring call counts per function etc. This will give you the most accurate data but will likely have relatively high impact on performance.

Alternatively, you can do statistical profiling which basically means you sample the stack on a timer instead of instrumenting everything. This can be much cheaper, even with high sampling rate! The downside of course is a loss of precision.

Even though it is surprisingly easy to do in Python, this stuff is still a bit much to put into an answer here. There is an excellent blog post by the Nylas team on this exact topic though.

The sampler below was lifted from the Nylas blog with some tweaks. After you start it, it fires an interrupt every millisecond and records the current call stack:

import collections
import signal

class Sampler(object):
    def __init__(self, interval=0.001):
        self.stack_counts = collections.defaultdict(int)
        self.interval = interval

    def start(self): 
        signal.signal(signal.VTALRM, self._sample)
        signal.setitimer(signal.ITIMER_VIRTUAL, self.interval, 0)

    def _sample(self, signum, frame):
        stack = []
        while frame is not None:
            formatted_frame = '{}({})'.format(
                frame.f_code.co_name,
                frame.f_globals.get('__name__'))
            stack.append(formatted_frame)
            frame = frame.f_back
        formatted_stack = ';'.join(reversed(stack))
        self.stack_counts[formatted_stack] += 1
        signal.setitimer(signal.ITIMER_VIRTUAL, self.interval, 0)

You inspect stack_counts to see what your program has been up to. This data can be plotted in a flame-graph which makes it really obvious to see in which code paths your program is spending the most time.

Upvotes: 9

Deepanshu Mishra
Deepanshu Mishra

Reputation: 22

Maybe make a text file and through your every program method just append some text referenced to it like "Method one executed". Run the web application like 10 times thoroughly as a viewer would and after this make a python program that reads the file and counts a specific parts of it or maybe even a pattern and adds it to a variable and outputs the variables.

Upvotes: -2

Related Questions