CODEWITHSUNDEEP
c++pythonmultithreadinguser-interfaceboost-python
eqzx
eqzx

Reputation: 5599

Multithreading Boost Python C++ Code in PySide

I have c++ code that I am wrapping with Boost Python. The idea is, I create a shared object that my python GUI can use to instantiate variables that wrap the c++ functionality

The c++ code does some heaving lifting, and I want to be able to make the wrapped object run concurrently so that the GUI doesn't block.

I compiled a Boost Python shared object wrapped with CMake like so:

find_package(Boost COMPONENTS system thread python REQUIRED)
find_package(PythonLibs REQUIRED)
include_directories(${Boost_INCLUDE_DIRS})
include_directories(${PYTHON_INCLUDE_DIRS})
link_directories(${Boost_LIBRARY_DIRS})
link_directories(${PYTHON_LIBRARIES})
set(Boost_USE_MULTITHREADED ON)
add_library(mynewlibinpython SHARED src/boost_python_wrapper_code.cpp)
target_link_libraries(mynewlibinpython ${Boost_LIBRARIES} ${PYTHON_LIBRARIES} mylibincpp)

I can instantiate the object in python fine, but it causes the GUI to block. Here is a generalized snippet of my python code:

from mynewlibinpython import *
class CppWrapper(QtCore.QObject):
    def __init__(self):
         super(CppWrapper,self).__init__()
         #some more initialization...

    @QtCore.Slot()
    def heavy_lifting_cpp_invocation():
         #constructor that takes a long time to complete
         self.cpp_object = CppObject()

and in an object that has a member function that is triggered when I press a button in the GUI:

class GuiObjectWithUnimportantName:
     def __init__(self):
          #inititalization

          self.cpp_thread = QThread()
          self.cpp_wrapper = CppWrapper()

     def function_triggered_by_button_press(self):
          self.cpp_wrapper.moveToThread(self.cpp_thread)
          self.cpp_thread.started.connect(self.cpp_wrapper.heavy_lifting_cpp_invocation)
          print "starting thread"       
          self.cpp_thread.start()
          print "Thread started"

So the cpp code will start doing its thing, and I'll also get the output basically immediately from both of the print statements ("starting thread" and "Thread started").

But the GUI is frozen. Does this have anything to do with the python GIL? I tried adding those macros into the boost_python_wrapper_code.cpp and recompiled, but it causes the python GUI to segfault immediately upon startup.

Upvotes: 4

Views: 922

Answers (1)

jdi
jdi

Reputation: 92627

Yes, I believe your problem is GIL related. Here is a simple example of what might be happening:

from PyQt4 import QtCore, QtGui
import time

class Window(QtGui.QDialog):

    def __init__(self):
        super(Window, self).__init__()
        self.resize(200,100)

        layout = QtGui.QVBoxLayout(self)

        layout.addWidget(QtGui.QLineEdit())

        self.button1 = QtGui.QPushButton("Sleep")
        self.button1.clicked.connect(self.go_safe)
        layout.addWidget(self.button1)

        self.button2 = QtGui.QPushButton("Crunch")
        self.button2.clicked.connect(self.go_GIL_buster)
        layout.addWidget(self.button2)

    def go_safe(self):
        t = QtCore.QThread(self)
        heavy = Heavy()
        heavy.moveToThread(t)
        t.started.connect(heavy.safe)
        print "starting thread"
        t.start()
        print "thread started"
        self.t1 = t
        self.heavy1 = heavy

    def go_GIL_buster(self):
        t = QtCore.QThread(self)
        heavy = Heavy()
        heavy.moveToThread(t)
        t.started.connect(heavy.GIL_buster)
        print "starting thread"
        t.start()
        print "thread started"
        self.t2 = t
        self.heavy2 = heavy

class Heavy(QtCore.QObject):

    def safe(self):
        print "Sleeping"
        time.sleep(10)
        print "Done"

    def GIL_buster(self):
        print "Crunching numbers"
        x = 2
        y = 500000000
        foo = x**y
        print "Done"


if __name__ == "__main__":
    app = QtGui.QApplication([])
    win = Window()
    win.show()
    win.raise_()
    app.exec_()

When you click the first button, the thread will do a sleep for 10 seconds, and you will notice that you can still type into the text field. A sleep releases the GIL so the main thread can continue to work.

When you click the second button, the thread does a heavy number operation. This acquires the GIL the entire time, and the GUI will lock until it is done.

If your boost extension isn't doing anything with python objects in any way, I think you can release the GIL for its operations. Then it won't block the GUI. If you are doing heavy operations on python objects, then you might have a problem here.

There is a tip on creating a scope-based GIL releasing class that you can use in any function to release the GIL for the duration of that function (when the helper instance gets destroyed)

class ScopedGILRelease
{
// C & D ----------------------------
public:
    inline ScopedGILRelease()
    {
        m_thread_state = PyEval_SaveThread();
    }

    inline ~ScopedGILRelease()
    {
        PyEval_RestoreThread(m_thread_state);
        m_thread_state = NULL;
    }

private:
    PyThreadState * m_thread_state;
};

And using it...

int foo_wrapper(int x)
{
    ScopedGILRelease scoped;
    return foo(x);
}

Someone else has some more examples of using this tip in this repo: https://bitbucket.org/wwitzel3/code/src/tip/nogil/nogil.cpp

Upvotes: 3

Related Questions