CODEWITHSUNDEEP
pythonpyside2qthread
JokerMartini
JokerMartini

Reputation: 6147

QThread managment tasks

How does qthread manage how many are being processed when I only have 4 threads max? For example say I have 100 long processing tasks. What happens when I for loop through my 100 tree view items and emit a Qthread to process the each task? Does it automatically determine I only have 4 threads and process the 100 threads within the 4 available?

Curious. Thanks

import os
import sys
import time
import random
from PySide2 import QtCore, QtGui, QtWidgets


class TaskThread(QtCore.QThread):
    
    finished = QtCore.Signal(object)

    def __init__(self, index):
        super(TaskThread, self).__init__()        
        self._index = index

    def runTask(self, task):
        self._task = task

    def run(self):
        t = random.randint(0,4)
        time.sleep(t)
        self.finished.emit(self._index)


class ExampleDialog(QtWidgets.QDialog):

    def __init__(self):
        super(ExampleDialog, self).__init__()
        
        self.itemModel = QtGui.QStandardItemModel()

        self.uiListView = QtWidgets.QListView()
        self.uiListView.setModel(self.itemModel)

        self.mainLayout = QtWidgets.QVBoxLayout(self)
        self.mainLayout.addWidget(self.uiListView)

        self.populateItems()


    def populateItems(self):
        self.threads = []
        for x in range(100):
            output = 'C:/Users/JokerMartini-Asus/Desktop/Trash/thumbs/IMG_409{}.jpg'.format(x)

            # Item
            item = QtGui.QStandardItem('{}'.format(x))
            item.setData(QtGui.QPixmap(), QtCore.Qt.DecorationRole)
            item.setData(output, QtCore.Qt.UserRole)
            self.itemModel.appendRow(item)

            mIndex = QtCore.QPersistentModelIndex(self.itemModel.indexFromItem(item))
            tt = TaskThread(mIndex)
            
            self.threads.append(tt)
            tt.start()
            tt.finished.connect(self.processFinished)


    def processFinished(self, index):
        if index.isValid():
            model = index.model()
            model.setData(index, '{} updated'.format(index.row()), QtCore.Qt.DisplayRole)


if __name__ == '__main__':
    pass
    app = QtWidgets.QApplication(sys.argv)
    window = ExampleDialog()
    window.resize(300,600)
    window.show()
    window.raise_()
    sys.exit(app.exec_())

Upvotes: 0

Views: 111

Answers (1)

musicamante
musicamante

Reputation: 48260

Note: I'm basing this answer only to empirical experimentation and basic knowledge of threads. I don't have enough knowledge on the lower implementation of threads to provide reliable resources of what I'm expressing, nor I can certify that this is valid in the same way on all platforms Qt supports (I'm on Linux).

There are two important things to consider when dealing with threading, even with Qt bindings:

  1. threading is not about "speeding up" things, but allowing computations to process when others are temporarily idle;
  2. the GIL only allows one operation at a time for each process, the fact that we perceive an apparent "parallelism" is due to the fact that each thread is temporarily releasing control to another one;

Fundamentally speaking, there's no absolute order, and in python there's no strict priority for a thread (due to the above reasons): processing is theoretically done [almost] in the order of thread starting, but the results may vary depending on how "fast" the processing is done for each thread, even for identical operations. As soon as a thread releases control to the GIL, the "next" thread gets it.

You could see a slightly more reliable result by adding the following changes to your code:

class TaskThread(QtCore.QThread):
    # ...
    def run(self):
        # always use the same timeout
        time.sleep(1)
        self.finished.emit(self._index)


class ExampleDialog(QtWidgets.QDialog):
    # ...
    def processFinished(self, index):
        while app.hasPendingEvents():
            app.processEvents()
            time.sleep(.05)
        self.uiListView.repaint()
        # ...

Note: the above code will block the UI, hasPendingEvents has been considered obsolete exactly due to the concurrency of threading and the usage processEvents is discouraged for the same reason; consider it only for educational purposes.

This means three things:

  • generally speaking, there's no "limit" to concurrent threads in python (since threads are not concurrent);
  • heavy duty computations are not actually parallel in python with threads;
  • most importantly, the number of "cores" is completely pointless when speaking about threads (especially in python): given the same situation, you'll get the same results for a 2-core CPU as you'd get for a 8-core CPU;

True parallelism can only be achieved through multiprocessing, but the problem is that processes, unlike threads, don't share memory. This doesn't mean that it's completely impossible: this link claims they achieved a proper multiprocessing support for Qt signal/slot mechanism, but I never tried it and I cannot say it actually works (especially considering that that link is quite old).

Note: you shouldn't connect the signals of a thread after it has started, especially if the thread can possibly return immediately (time.sleep(0)); move the tt.finished.connect line before tt.start().

Upvotes: 1

Related Questions