Tkinter Performance Issues - Follow Up

Question

I am trying to improve the performance of an application I'm working on. Following up from my previous questions: here and here, and after profiling my code, the issue seems to mainly lie within Tkinter.

My issue is that displaying a large number of tkinter widgets (~850) causes a large dip in performance - it takes several seconds to display the widgets and the UI is not fully responsive until every widget is displayed.

I will try to walk you through my entire thought process in trying to find where the issue lies.

---

Preliminary

Now the actual code is very long so I will try to explain more or less the workings of each of the main culprits.

MenuView.py: contains the entire UI

• create_pages is called by the grid method and is responsible for creating the different pages on the UI and calling their respective grid methods.
• The grid method of each 'page' is the exact same, detailed below.
• Note that a 'page' is a script containing a single class which inherits from a Frame

tkButtons.py and tkFrames.py: contain classes of customised tkinter widgets

• The fetch method in one of the custom frames is detailed further below:

---

Profiling the Application

Here is the output when profiling the entire application with cProfile

         1134871 function calls (1117361 primitive calls) in 46.335 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   1144/1    0.074    0.000   46.342   46.342 {built-in method builtins.exec}
        1    0.000    0.000   44.213   44.213 __init__.py:1281(mainloop)
        1   41.090   41.090   44.213   44.213 {method 'mainloop' of '_tkinter.tkapp' objects}
  801/784    0.008    0.000    3.132    0.004 __init__.py:1700(__call__)
11147/11135    3.083    0.000    3.085    0.000 {method 'call' of '_tkinter.tkapp' objects}
      702    0.008    0.000    2.074    0.003 tkButtons.py:37(__init__)
      702    0.007    0.000    2.043    0.003 tkButtons.py:9(__init__)
      875    0.001    0.000    1.687    0.002 __init__.py:1478(configure)
      887    0.003    0.000    1.687    0.002 __init__.py:1466(_configure)
        1    0.000    0.000    1.265    1.265 MenuView.py:68(grid)
        1    0.000    0.000    1.265    1.265 MenuView.py:47(create_pages)
    715/1    0.007    0.000    1.193    1.193 <frozen importlib._bootstrap>:966(_find_and_load)
    712/1    0.005    0.000    1.193    1.193 <frozen importlib._bootstrap>:936(_find_and_load_unlocked)
    967/2    0.001    0.000    1.192    0.596 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
    690/3    0.006    0.000    1.190    0.397 <frozen importlib._bootstrap>:651(_load_unlocked)
    562/1    0.003    0.000    1.190    1.190 <frozen importlib._bootstrap_external>:672(exec_module)
        1    0.000    0.000    1.190    1.190 UI_Controller.py:1(<module>)
1643/1248    0.001    0.000    1.144    0.001 {built-in method builtins.next}
       22    0.000    0.000    1.141    0.052 tkFrames.py:196(fetch)
       20    0.000    0.000    1.091    0.055 __init__.py:747(callit)
   469/30    0.001    0.000    1.065    0.035 {built-in method builtins.__import__}
3151/1079    0.005    0.000    0.951    0.001 <frozen importlib._bootstrap>:997(_handle_fromlist)
        1    0.000    0.000    0.784    0.784 Model.py:6(<module>)
        1    0.000    0.000    0.776    0.776 LeagueController.py:5(<module>)
        1    0.000    0.000    0.772    0.772 LeagueGenerator.py:1(<module>)
        1    0.000    0.000    0.739    0.739 FreeAgentGenerator.py:1(<module>)
        1    0.000    0.000    0.737    0.737 PlayerGenerator.py:1(<module>)
        1    0.000    0.000    0.731    0.731 StatGenerator.py:1(<module>)

I am happy with the speed of loading data into the UI, but not with how quickly this data is displayed.

Help in really understanding what this output means is also appreciated.

---

1. The Buttons

The most time is spent in the initialisers of two classes in the TkButtons.py file. Here is a simplified outline of how it looks:

class DynamicButton(Button):
    def __init__(self, parent, **kwargs):
        super().__init__(parent, **kwargs)
        self.font = self.get_font() # method to get the font of the button
        self.bind("<Configure>", self._on_configure)


    def _on_configure(self, event):       
        text = self.cget("text")
        size = self.font.actual("size")
        width, height = event.width, event.height
        while size > 1 and self.font.measure(text) >= width:
            size -= 1
            self.font.config(size=size)


class HoverButton(DynamicButton):
    def __init__(self, parent, bind=False, **kwargs):
        super().__init__(parent, **kwargs)
        self.bind("<Enter>", self.on_enter)
        self.bind("<Leave>", self.on_leave)
        if bind:
            self.bind("<Button-1>", self.on_click)

    def on_enter(self, event=None):
        # change bg color of button

    def on_leave(self, event=None):
        # change bg color of button back to original

    def on_click(self, event):
        # change bg color of button

My code creates HoverButton objects which inherit from the DyanmicButton class. Now I am aware that the DynamicButton is one of the problems but removing it still leaves performance issues elsewhere so I will continue on.

In total, there are 844 buttons to display (from the tkButtons.py file) and I have no idea how to make it any faster.

---

2. Producing an MCVE

To illustrate the problem I'm facing, here is some sample code which mimics the behaviour of the application:

from tkinter import Tk, Button, Frame
from random import randint as rand

class MenuView(Frame):
    def __init__(self, parent, **kwargs):
        super().__init__(parent, **kwargs)
        self.parent = parent
        self.rows, self.cols = 30, 29
        self.headers = [0]*self.cols
        self.data = [[rand(25, 99) for _ in range(self.cols)] for _ in range(self.rows)]
        self.btns = [[None for _ in range(self.cols)] for _ in range(self.rows)]

    def grid(self, **kwargs):
        super(Frame, self).grid(**kwargs)
        self.a = self.display_data()
        self.fetch()

    def fetch(self):
        try: next(self.a)
        except StopIteration: return

    def display_data(self):
        for row in range(self.rows):
            for column in range(self.cols):
                self.btns[row][column] = Button(self, text=self.data[row][column],
                                              command=lambda column_index=column:
                                              self.sort(column_index), bg='red')                
                self.btns[row][column].grid(row=row, column=column)
            if not row % 5:
                self.parent.after(10, self.fetch)
                yield

    def modify_cells(self):
        for row in range(self.rows):
            for column in range(self.cols):
                self.btns[row][column].config(text=self.data[row][column],
                                              command=lambda column_index=column:
                                              self.sort(column_index))

    def sort(self, column_index):
        self.data.sort(key=lambda x: x[column_index])
        self.modify_cells()                

win = Tk()
MenuView(win).grid(sticky='nesw')
win.mainloop()

Profiling this code sample, for comparison, gives:

         52906 function calls (52840 primitive calls) in 49.029 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      7/1    0.000    0.000   49.029   49.029 {built-in method builtins.exec}
        1    0.000    0.000   49.029   49.029 page performance.py:1(<module>)
        1    0.000    0.000   48.706   48.706 __init__.py:1281(mainloop)
        1   44.385   44.385   48.706   48.706 {method 'mainloop' of '_tkinter.tkapp' objects}
        7    0.000    0.000    4.322    0.617 __init__.py:1700(__call__)
     2621    4.236    0.002    4.236    0.002 {method 'call' of '_tkinter.tkapp' objects}
        1    0.000    0.000    0.295    0.295 __init__.py:2003(__init__)
        1    0.295    0.295    0.295    0.295 {built-in method _tkinter.create}
        7    0.000    0.000    0.100    0.014 page performance.py:18(fetch)
        7    0.000    0.000    0.100    0.014 {built-in method builtins.next}
        7    0.005    0.001    0.100    0.014 page performance.py:22(display_data)
        6    0.000    0.000    0.086    0.014 __init__.py:747(callit)
      870    0.001    0.000    0.070    0.000 __init__.py:2350(__init__)
      871    0.006    0.000    0.069    0.000 __init__.py:2286(__init__)
     1742    0.009    0.000    0.025    0.000 __init__.py:1315(_options)
      871    0.002    0.000    0.024    0.000 __init__.py:2209(grid_configure)
        1    0.000    0.000    0.014    0.014 page performance.py:13(grid)

Now, it says the grid method takes 0.014s but it takes well over 3 seconds for all the buttons to show up on my system. So this has thrown me off quite a bit.

---

3. Application Structure

Nevertheless, this suggests to me that the issue may lie with the structure of my application:

Apart from the tkinter issues, the output is showing that a lot of time is taken up in the first lines of UI_Controller.py, Model.py, LeagueController.py, LeagueGenerator.py, FreeAgentGenerator.py, PlayerGenerator.py and StatGenerator.py. The lines they refer to are literally the start to each file - the imports. Is there any way to reduce this?

Here is my file structure (note that all files using tkinter will import constants from the tkConstants.py file):

.
+-- run.py
+-- _main
    |   +-- _controllers
        |   +-- LeagueController.py
        |   +-- UI_Controller.py
    |   +-- _data_models
        |   +-- Model.py

+-- _misc
    |   +-- _tk
        |   +-- tkConstants.py  # File containing constants
        |   +-- _widgets
            |   +-- tkFrames.py  # Uses data from tkConstants.py
            |   +-- tkButtons.py  # Uses data from tkConstants.py

+-- _setup
|   +-- LeagueGenerator.py
|   +-- StatGenerator.py
|   +-- PlayerGenerator.py
|   +-- FreeAgentGenerator.py

+-- _ui
|   +-- _game_ui
    |   +-- _ui_pages
        |   +-- page1.py
        |   +-- page2.py
                   . 
                   . 
                   . 
        |   +-- page7.py

    |   +-- MenuView.py  # Imports ui page files from _ui_pages folder  

|   +-- _main_menu
    |   +-- main_menu.py  # Imports load_page.py and new_page.py
    |   +-- load_page.py  # Imports MenuView.py, Model.py and all files from _setup
    |   +-- new_page.py  # Imports MenuView.py, Model.py and all files from _setup

(For reference, there are a total of 54 files in the application)

This doesn't make much sense to me but it ties in with why so much time is being spent importing scripts.

I'm not too informed on how python imports scripts from folders and hence unsure how I can improve performance if the issue lies here.

---

4. Python Inheritance Overhead

Finally, I tried combining the outline of the tkButtons.py file above with the code sample above:

Profiling that code then gives:

         110635 function calls (110569 primitive calls) in 63.940 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      7/1    0.000    0.000   63.940   63.940 {built-in method builtins.exec}
        1    0.000    0.000   63.940   63.940 page performance.py:1(<module>)
        1    0.000    0.000   63.615   63.615 __init__.py:1281(mainloop)
        1   63.331   63.331   63.615   63.615 {method 'mainloop' of '_tkinter.tkapp' objects}
        1    0.288    0.288    0.288    0.288 {built-in method _tkinter.create}
      956    0.010    0.000    0.284    0.000 __init__.py:1700(__call__)
        7    0.000    0.000    0.162    0.023 page performance.py:54(fetch)
        7    0.000    0.000    0.162    0.023 {built-in method builtins.next}
        7    0.005    0.001    0.162    0.023 page performance.py:58(display_data)
        6    0.000    0.000    0.147    0.025 __init__.py:747(callit)
      870    0.004    0.000    0.128    0.000 page performance.py:20(__init__)
      870    0.003    0.000    0.097    0.000 page performance.py:5(__init__)
        1    0.000    0.000    0.015    0.015 page performance.py:49(grid)

Once again, the time taken for grid to run is not actually what is observed. It seems as if the double inheritance has affected the performance although not as much as I expected it to.

However, in my simplified test, the method bodies in each button are all empty.

---

Conclusion

Other than the above, I'm completely stumped as to where the bottleneck lies.

I am aware that the issues I've described are not as extreme for some other users but I would still like to find a workaround.

Once again, the issue is not (from what I can see, loading takes 0.223s) in how I load data into the UI, but rather creating and displaying the widgets to show the data.

If you have got this far, thank you and I do apologise for the (very) lengthy question but I've been stuck on this problem for a month and a half now.

As always, any help/guidance is greatly appreciated.

Answer 1

I would guess the bottleneck is simply all of the computations that tkinter must do to render the widgets. One problem is just the raw number of widgets. You've created nearly 900 widgets which is a lot for tkinter to handle.

If you are concerned about rendering speed, and unless you feel like you absolutely must use button widgets, I recommend using a single canvas where you draw the cells as items on the canvas. You can attach your own bindings to the items on the canvas to make them behave like buttons.

When I run your code as written there is a very noticeable several-second delay to render all the buttons. When I modify your code to use a canvas, the display comes up nearly instantaneously.