Python 3 Tkinter Treeview Slow Performance while Multithreaded

Question

Tl;dr: 1500 inserts to a Treeview widget from another thread have been taking 40-340 seconds to complete, but take only 1.2-1.7 seconds to complete if inserted while on the main thread or while the root window is so small that Treeview can't be seen.

I'm working on a little project involving neural networks identifying images for a game, and I'm trying to figure out why my results Treeview updates so slowly after identifying the images. The code I have below is an MCVE of the general issue I'm running into.

My application currently trains the networks, makes predictions based on the training models, then when displaying the results of those predictions takes longer than both the training and predictions take to display 1500 results. In both my application and this example I've found that in general how long the populate button takes to complete and display all results seems to be directly proportional to how tall the window is (or how tall the Treeview widget is), with this example ranging between 40 and 340 seconds to display all 1500 rows of output on my current machine. I tried using cProfile and pstats to determine what more specifically is causing the delay, however I'm still inexperienced with them and although I know that roughly 99% of the time is spent on '{method 'call' of '_tkinter.tkapp' objects}' and '{method 'globalsetvar' of '_tkinter.tkapp' objects}', I don't know what either of these are, or what to do with that information.

However, I've found that if I start the worker function while the window is so small that the Treeview can't be displayed, then it takes roughly 1.2 - 1.7 seconds to display all results. This can be visibly seen by watching the progress bar in my example and seeing how it progresses much more quickly the smaller you make the window. Due to the fact that it can display all results in this amount of time demonstrates (at least to me) that the vast majority of the time spent with the Treeview visible while inserting results is spent rendering the text over and over again and updating the height of the scrollbar. To that end, I've been trying to find a way to insert a large quantity of rows at once or at least without rendering the Treeview again after every change, but haven't found anything that seems to be able to do that.

While trying to create this MCVE, I found that it takes an equally short amount of time (~1.5 seconds) to display all results if I call the add_entries function on the main thread instead of calling it on another thread. While I think this could be a viable solution, I'm curious if there's a better solution available while trying to get more information about a problem I've so far struggled to find much about.

The closest I've found so far is this discussion talking about how with a similar module (GTK3) someone had a similar issue and a solution was to set the Treeview to a fixed height model, however I can't find any information about a similar option being available to regular tkinter Treeview widgets as opposed to just GTK3, and was curious if such an option exists in tkinter or if that's exclusive to other modules, or if there's a better solution out there that I haven't thought of entirely.

    import random, threading
    import cProfile, pstats
    from tkinter import *
    from tkinter.font import Font
    from tkinter.ttk import Progressbar, Treeview
    from string import printable
    
    COLS = list(range(10))
    ROWS = 1500
    
    def __main__():
        root = Tk()
        program_window = App(root)
        try:
            root.destroy()
        except TclError:
            pass
    
    class App(Frame):
        def __init__(self, parent=None):
            self.parent = parent
            
            self.bar_int = IntVar(value=0)
            self.tree = Treeview(self.parent, columns=COLS, show="headings")
            self.vsb = Scrollbar(self.parent, orient='vertical', command=self.tree.yview)
            self.bar = Progressbar(self.parent, variable=self.bar_int)
            self.btn = Button(self.parent, text="Populate", command=self.add_entries)
            
            for col in COLS:
                self.tree.heading(col, text=str(col))
                self.tree.column(col, width=Font().measure(str(col)))
            
            self.tree.grid(row=0, column=0, sticky='nsew', columnspan=2)
            self.vsb. grid(row=0, column=2, sticky='nsew')
            self.bar. grid(row=1, column=0, sticky='nsew')
            self.btn. grid(row=1, column=1, sticky='nsew', columnspan=2)
            
            self.parent.columnconfigure(0, weight=1)
            self.parent.rowconfigure(0, weight=1)
            self.parent.geometry('300x200')
            
            self.parent.mainloop()
        
        def add_entries(self):
            worker = threading.Thread(target=self.add_entries_worker)
            worker.start()
        
        def add_entries_worker(self):
            self.tree.delete(*self.tree.get_children())
            self.bar.configure(maximum=ROWS)
            with cProfile.Profile() as profile:
                for i in range(ROWS):
                    self.bar_int.set(i)
                    li = [random.sample(printable, 10) for i in COLS]
                    self.tree.insert('', 'end', values=li)
                ps = pstats.Stats(profile)
                ps.print_stats()
    
    if __name__ == "__main__":
        __main__()

Update: at hussic's suggestion and after talking to a friend about the issue, I looked into root.after() and Queue. I was able to get my current code working to a degree with hussic's suggestion, but it still seems to be unstable and I'm not certain if I've done everything as I should. Below is a new refresher() function I've added, along with the modifications to the add_entries() function. The add_entries_worker() just had all its references to the tkinter widgets removed and its insert to tree replaced with appending to self.data_queue.

    def refresher(self):
        # while loop so I can pop data out of the list one at a time rather than
        # for loop and potentially delete unprocessed data when clearing the list
        while self.data_queue:
            i, item = self.data_queue.pop(0)
            self.bar_int.set(i)
            self.tree.insert('', 'end', values=item)
        
        if self.btn['state'] == 'disabled':
            self.parent.after(100, self.refresher)
    
    def add_entries(self):
        self.btn.configure(state=DISABLED)
        self.tree.delete(*self.tree.get_children())
        self.bar.configure(maximum=ROWS)
        self.refresher()
        
        worker = threading.Thread(target=self.add_entries_worker)
        worker.start()

Answer 1

I added a class for tkinter thread, Tkworker, which use Queue and after(). Work to be done is divided in chunk (see: self.chunksize=500). Print something to terminal to check order.

import random, threading, queue
from tkinter import Tk, TclError, Frame, IntVar, Scrollbar, Button
from tkinter.font import Font
from tkinter.ttk import Progressbar, Treeview
from string import printable

COLS = list(range(10))
ROWS = 1500


def __main__():
    root = Tk()
    program_window = App(root)
    try:
        root.destroy()
    except TclError:
        pass


class Tkworker:
    Empty = queue.Empty

    def __init__(self, root, producer, consumer, ms=200):
        self.root = root
        self.consumer = consumer
        self.ms = ms
        self.queue = queue.Queue()  # type: queue.Queue
        self.thread = threading.Thread(target=producer)

    def start(self):
        self.stop = False
        self.thread.start()
        self._consumer_call()

    def put(self, item):
        self.queue.put(item)

    def get(self):
        return self.queue.get(False)

    def _consumer_call(self):
        self.consumer()
        if not self.stop:
            self.root.after(self.ms, self._consumer_call)


class App(Frame):

    def __init__(self, parent=None):
        self.parent = parent

        self.bar_int = IntVar(value=0)
        self.tree = Treeview(self.parent, columns=COLS, show="headings")
        self.vsb = Scrollbar(self.parent, orient='vertical', command=self.tree.yview)
        self.bar = Progressbar(self.parent, variable=self.bar_int)
        self.btn = Button(self.parent, text="Populate", command=self.start_entries)

        for col in COLS:
            self.tree.heading(col, text=str(col))
            self.tree.column(col, width=Font().measure(str(col)))

        self.tree.grid(row=0, column=0, sticky='nsew', columnspan=2)
        self.vsb. grid(row=0, column=2, sticky='nsew')
        self.bar. grid(row=1, column=0, sticky='nsew')
        self.btn. grid(row=1, column=1, sticky='nsew', columnspan=2)

        self.parent.columnconfigure(0, weight=1)
        self.parent.rowconfigure(0, weight=1)
        self.parent.geometry('300x200')

        self.parent.mainloop()

    def start_entries(self):
        self.chunksize = 500
        self.tkwr = Tkworker(self.parent, self.worker, self.add_entries, ms=100)
        self.tree.delete(*self.tree.get_children())
        self.bar.configure(maximum=ROWS)
        self.num = 0
        self.tkwr.start()

    def worker(self):
        chunk = []
        for i in range(ROWS):
            chunk.append([random.sample(printable, 10) for _ in COLS])
            if  i % self.chunksize == 0:
                self.tkwr.put(chunk)
                chunk = []
        if chunk:
            self.tkwr.put(chunk)
        print('worker end')

    def add_entries(self):
        try:
            chunk = self.tkwr.get()
            for li in chunk:
                self.tree.insert('', 'end', values=li)
            self.num += len(chunk)
            self.bar_int.set(self.num)
        except self.tkwr.Empty:
            if self.num == ROWS:
                self.tkwr.stop = True
                print('stop')


if __name__ == "__main__":
    __main__()