CODEWITHSUNDEEP
pythonperformancepyqt
Cesar
Cesar

Reputation: 717

File copy too slow when pyqt is involved

I am using this code (very simplified version of original but problem remains) to copy a file:

def copyfileobj_example(source, dest, buffer_size=1024*1024):
    while 1:
        copy_buffer = source.read(buffer_size)
        if not copy_buffer:
            break
        dest.write(copy_buffer)

If I call the function without pyqt the files are copied realy fast, but when I call it inside a simple pyqt window, the copy is three times slower.

Copy fast a huge number of files is the main point of the application, I assume that including the gui will slow things a little, but no make it three times slower!! and running the copy function using threads or multiprocess do not cause satisfactory improvements.

This is just as is? could you recommendme something to solve this performance issue?

EDIT: There is a gist with my actual copy code, running with and without PyQT

Upvotes: 3

Views: 506

Answers (2)

ekhumoro
ekhumoro

Reputation: 120598

Since I am unable to get the linked code in the question to work (it just hangs and uses 100% CPU), I will post a saner example for the purposes of testing.

Using the test case below, I get the following output when copying a 400MB file (three runs):

$ python copy_test.py
2.9546546936035156
2.9658050537109375
$ python copy_test.py
3.226983070373535
3.192814826965332
$ python copy_test.py
2.935734748840332
2.8552770614624023

As you can see, there is no significant difference. For clarity, this was using the following setup on Linux:

Python 3.5.0, Qt 5.5.0, PyQt 5.5

and I get similar results with all combinations of Python 2.7/3.5, PyQt 4/5.

Here is the test case:

import sys
import time
import os

SRC_FILE = '/home/tmp/source/test/test.zip'
DEST_FILE = '/home/tmp/source/test/test-copy.zip'

def copy_file(src, dst=[], progress=None, only_new_file=True):
    size = 1024 * 1024
    with open(src, 'rb') as s, open(dst[0], 'wb') as d:
        while 1:
            copy_buffer = s.read(size)
            if not copy_buffer:
                break
            d.write(copy_buffer)

if __name__ == '__main__':

    initTime = time.time()
    copy_file(SRC_FILE, [DST_FILE], only_new_file=False)
    print (time.time() - initTime)

    time.sleep(5)

    from PyQt5.QtWidgets import QApplication, QWidget
#     from PyQt4.QtGui import QApplication, QWidget

    app = QApplication(sys.argv)

    w = QWidget()
    w.resize(250, 150)
    w.move(300, 300)
    w.setWindowTitle('Simple')
    w.show()

    initTime = time.time()
    copy_file(SRC_FILE, [DST_FILE], only_new_file=False)
    print (time.time() - initTime)

    sys.exit(app.exec_())

Upvotes: 2

Aaron Digulla
Aaron Digulla

Reputation: 328564

This is probably an effect of the GIL. With PyQt running in the UI thread, it "steals" the GIL every time it has to handle events. That means your loop above stops every time. This also happens when you're running in another thread; the lock is global.

Workarounds:

  1. Use a larger buffer. The C layer of Python isn't affected by the GIL, so if you copy larger amounts of data, the commands in the loop are executed less often.
  2. Execute an external command to do the copying (possibly another Python process).
  3. Use Qt's IO classes to copy the file since the aren't affected by the GIL either (Kudos to ekhumoro for the idea).
  4. Write a piece of code in C which transfers the data.
  5. Use a version of Python which doesn't have a GIL like IronPython or Jython.

Upvotes: 5

Related Questions