How to profile performance in a QGIS Python plugin?

Question

Is it possible to use kernprof.py, line_profiler.py, or something similar to profile a QGIS plugin? I can't run the plugin outside of QGIS because the plugin requires state from QGIS & will make calls to the QGIS API.

It seems like I might be able to modify the plugin's initializer to call kernprof, to call back to the plugin and pass the state all the way through, but I can't wrap my head around it.

Does anyone have experience with running a Python profiler from inside another tool?

Answer 1

It's possible to use line_profiler while running your script inside QGIS.

You need to import it inside the main file of your plugin along your other imports, then add profile = line_profiler.LineProfiler() before your main class, add the decorator @profile just before your main function to profile and finally add profile.print_stats(stream=stream) just before the return of the function.

I suppose there is other ways to do it, but it's the way I found that works good enough for me.

Below is an example for a Processing plugin:

import os

import line_profiler
profile = line_profiler.LineProfiler() 

class processingScriptExample(QgsProcessingAlgorithm):
    
    INPUT_directory = 'INPUT_directory'
    
    def initAlgorithm(self, config):
        self.addParameter(QgsProcessingParameterNumber(self.INPUT_directory,
                                                       self.tr('Output directory'),
                                                       QgsProcessingParameterFile.Folder))
    
    @profile
    def processAlgorithm(self, parameters, context, feedback):
        directory = self.parameterAsInt(parameters, self.INPUT_directory, context)
        
        ls = []
        for ii in range(1000000):
            ls.append(ii)
        
        ls = [ii for ii in range(1000000)]
        
        path_profiling = os.path.join(directory, "line_profiling.txt")
        with open(path_profiling, 'w') as stream:
            profile.print_stats(stream=stream)

        return {'Profiling file': path_profiling}

The resulting file:

Timer unit: 1e-07 s

Total time: 1.31260 s
File: C:\OSGeo4W\profiles\default/python/plugins\test\algo_test.py
Function: processAlgorithm at line 70

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    70                                               @profile
    71                                               def processAlgorithm(self, parameters, context, feedback):
    72         1        248.0    248.0      0.0          directory = self.parameterAsInt(parameters, self.INPUT_directory, context)
    73                                           
    74         1          8.0      8.0      0.0          ls = []
    75   1000001    5054594.0      5.1     38.5          for ii in range(1000000):
    76   1000000    6633146.0      6.6     50.5              ls.append(ii)
    77                                                   
    78         1    1418416.0 1418416.0     10.8          ls = [ii for ii in range(1000000)]
    79                                                   
    80         1        561.0    561.0      0.0          path_profiling = os.path.join(directory, "line_profiling.txt")
    81         1      19001.0  19001.0      0.1          with open(path_profiling, 'w') as stream:
    82                                                       profile.print_stats(stream=stream)
    83                                           
    84                                                   return {"Profiling file":path_profiling}

Answer 2

I used a simpler way to profile my plugin using cProfile. In the constructor of main class of plugin (that is returned in classFactory), I used this code:

 self.pr = cProfile.Profile()
 self.pr.enable()

and in unload method of the class or any where some one needs print the profile stats:

    self.pr.disable()
    s = io.StringIO()
    sortby = SortKey.CUMULATIVE
    ps = pstats.Stats(self.pr, stream=s).sort_stats(sortby)
    ps.print_stats() 

remember to use following code for imports:

import cProfile, pstats, io
from pstats import SortKey