How do I make pythons Queue.Queue work faster?

Question

I have been working on a tkinter-GUI from which I start long-running external scripts whose outputs are then written into the GUI. In order for my GUI not to freeze while the scripts are running, I use the threading-module.

This worked in principle, but I then ran into the problem that my GUI would often crash. Because the scripts gave it output faster than it could display it, I suppose.

To solve this problem, I now write the output of my scripts into a queue, using the Queue-module. Now it works almost as intended, but if I have many processes (>20) run at the same time, all of them start becoming very slow, or at least the outputs are displayed with a time delay. I guess this is because the queue "can't keep up" with the output of the scripts.

This leads me to my question: Is there a way to make make the queue work faster, or alternatively is there a way to make my program work faster otherwise?

Here is a minimal example of my code. Instead of the external scripts I just let the program continuously count up:

import sys, os
import Tkinter as tk
from threading import Thread
import time
import Queue


class GUI:
    def __init__(self,master):
        self.master = master
        master.title('Threading Test')

        self.thread_queue = Queue.PriorityQueue()#Queue where all the outputs are lined up
        self.runprocess = [0 for i in range(30)]#These will be the theads, later

        #Dropdown menu that lets you select the output window
        self.options = tk.StringVar()
        self.options.set('1')
        self.numbers = [str(q) for q in range(1,30)]
        self.Option = tk.OptionMenu(master,self.options,*self.numbers,command=self.Select_Window)
        self.Option.pack(fill='x')

        #Button that starts function (in a thread)
        self.button = tk.Button(master,text='start',command = self.run_thread)
        self.button.pack()

        #Output windows
        self.Output_Frame = tk.Frame(master,width=800,height=100)
        self.Output_Frame.pack(fill='both',expand=1)
        self.Output_Frame_Pages = [0 for i in range(30)]
        self.Output_Fields = [0 for i in range(30)]
        self.Output_Scroll = [0 for i in range(30)]
        for q in range(len(self.Output_Frame_Pages)):
            self.Output_Frame_Pages[q] = tk.Frame(self.Output_Frame)
            self.Output_Frame_Pages[q].place(in_=self.Output_Frame,x=0,y=0,relwidth=1,relheight=1)
            self.Output_Fields[q] = tk.Text(self.Output_Frame_Pages[q],bg='white')
            self.Output_Fields[q].pack(fill='both',expand=1,side='left')
            self.Output_Fields[q].configure(state='disabled')
            self.Output_Scroll[q] = tk.Scrollbar(self.Output_Frame_Pages[q],command=self.Output_Fields[q].yview)
            self.Output_Scroll[q].pack(side='left',fill='y')
            self.Output_Fields[q]['yscrollcommand'] = self.Output_Scroll[q].set
        self.Output_Frame_Pages[0].lift()

        #Function that checks if there is something in the queue and then writes the output
        self.master.after(1,self.doWork())

    #Function that lets you chose the output window
    def Select_Window(self,ddmvar):
        self.Output_Frame_Pages[int(self.options.get())-1].lift()

    #Function that writes output
    def Write_Output(self,Message,Window):
        self.Output_Fields[Window-1].configure(state='normal')
        self.Output_Fields[Window-1].insert(tk.END,str(Message)+'\n')
        self.Output_Fields[Window-1].see(tk.END)
        self.Output_Fields[Window-1].configure(state='disabled')

    #Function that "does stuff"
    def run(self):
        i = 0
        aux = int(self.options.get())
        while True:
            i+=1
            self.thread_queue.put((i,aux))
            #print 'running'

    #Function that calls "run" in a thread, so GUI does not freeze
    def run_thread(self):
        aux = int(self.options.get())
        self.runprocess[aux-1] = Thread(target=self.run)
        self.runprocess[aux-1].daemon = True
        self.runprocess[aux-1].start()

    #Function that checks if there is something in the queue an then writes the output
    def doWork(self):
        try:
            self.Write_Output(*self.thread_queue.get(0))
            self.master.after(1,self.doWork)
        except Queue.Empty:
            self.master.after(1,self.doWork)



root = tk.Tk()
gui = GUI(root)
root.mainloop()

Another problem I noticed is that my threads seem to continue, even if I close the GUI. This can be seen if I let my function not only write to the GUI but also let my program print something. I don't know if this is related to the problem described here, though.

Thanks in advance for your help!

Edit: In case it matters: I run python 2.7 on Linux.

Answer 1

You are introducing a large amount of overhead when you use after to call a function every millisecond, but in each call you only pull one item off of the queue. What you are doing is just about the most inefficient solution possible

Instead, your function should pull more than one item off of the queue at a time. The exact number depends on many factors, and sometimes the best way to know the number is to experiment. You might also want to consider using a value greater than 1ms to pull items from the queue. At 1ms between calls, it doesn't provide tkinter with much time to process all of the other events it needs to process.

For example, you might have after call the function every 10ms but you pull 100 items off at a time. Or, you might call it every 100ms and completely drain the queue.