Investigating memory leak in python Kivy app with memory_profiler

Question

I'm not new with python, quite beginner with Kivy though. And never ever tried to catch a memory leak.

I went pretty far with developing my python Kivy app when accidentally noticed (using Windows task manager that the app constantly taking more and more memory when no user interaction happens.

Besides main app this program has 2 threads. One of them seems not guilty as I could use app without it and memory leak problem still exist. The other thread however seem to be guilty.

Here is what I did.

Googled for python memory leak topics.
Youtubed the same.

I certainly found something to start with = memory_profiler, however I didn't find an good guide how to do it, how to find the leak. All the info tells about parameters, options, but not the methods to find the leak. Particularly that all examples everywhere are focused on functions, when Kivy app is an object oriented one.

Perhaps question #1: is there a very good guide on how to find memory leak in python object oriented app or ideally for Kivy app.

I tried to apply examples of working with memory_profiler to methods of my classes.

Knowing my app I suspected a thread that works in background for a memory leak. Then I took all internals of while loop that does the thread run and put it into a method so that it could be decorated with "@profile".

def run(self):
    while True:
        self.loop_internals()

@profile
def loop_internals(self):
    start_time = time.time()        
    for processor_value in self.processors.values():
        name = processor_value['name']
        if not processor_value['hardware_simulation'] and self.bus:
            try:
                hw_data = processor_value['hardware_processor'].getValue(processor_value)
                processed_data = processor_value['measurement_processor'].process(hw_data)
            except Exception as e:
                Logger.debug(f"ERROR: Device '{name}' / {processor_value['hardware_address']} got exception: {e}")
        elif not processor_value['hardware_simulation'] and not self.bus:
            processed_data = processor_value['measurement_processor'].process(0)
        else:
            hw_data = processor_value['hardware_processor'].getValue(processor_value)
            Logger.debug(f"data: Data from device '{processor_value['device']}' processor '{hw_data}'.")
            #processed_data = processor_value['measurement_processor'].process(hw_data)
            processor_value['measurement_processor'].process(hw_data)
            print(f" ************** processor_value {sys.getsizeof(processor_value)} ")
            print(f" ************** self.processors {sys.getsizeof(self.processors)} ")
            #Logger.debug(f"data: Data after processor '{processor_value['type']}' = '{processed_data}'.")
        
        self.notify_device_observers(processor_value['device'])

Then run it like:

mprof run --multiprocess main.py

Here is an example of output:

Line #    Mem usage    Increment  Occurrences   Line Contents

61    188.4 MiB      0.0 MiB           9                   hw_data = processor_value['hardware_processor'].getValue(processor_value)
62    188.4 MiB      0.0 MiB           9                   Logger.debug(f"data: Data from device '{processor_value['device']}' processor '{hw_data}'.")
64    188.4 MiB      0.1 MiB           9                   processor_value['measurement_processor'].process(hw_data)
65    188.4 MiB      0.1 MiB           9                   print(f" ************** processor_value {sys.getsizeof(processor_value)} ")
66    188.4 MiB      0.0 MiB           9                   print(f" ************** self.processors {sys.getsizeof(self.processors)} ")

************** processor_value 464
************** self.processors 272

Size of processor_value & self.processors NOT growing.

self.processors - is a dictionary and not growing in size.

processor_value['measurement_processor'] is a reference to a class - all the time the same.

.process(hw_data) calling a method of that class. According to memory_profiler isn't leaking.

I also decorated method "process" of that class with "@profile" and I get all zeros in memory increment for it in profile console printout. So the method "process" according to memory_profiler does not leak memory.

Naturally since I'm not sure I even using right methods to find the leak - I'm not confident that I'm on a right track with this find. But if I'm on the right track that - and increment 0.1Mib that profiler shows is a catch - then I still can not understand how come!?

processor_value isn't growing (and it shouldn't it is fixed dictionary)

method .process doesn't do memory leak (tested with profiler)

And so if I'm on the right track with methodology and if I did find the leak then how come this do the leak??? It seems so innocent call... Though that this loop runs multiple times a second and 0.1-0.2 memory increment could easily explain observation of constant increase of memory consumption by the app.

Fun fact! If I comment out a print, then profiler might detect 0.2MiB increase by my suspect:

Line #    Mem usage    Increment  Occurrences   Line Contents
64    199.9 MiB      0.2 MiB           9                   processor_value['measurement_processor'].process(hw_data)
65                                                         #print(f" ************** processor_value {sys.getsizeof(processor_value)} ")

Investigating memory leak in python Kivy app with memory_profiler

Answers (1)

Related Questions