CODEWITHSUNDEEP
pythonmultithreadingqtpysidecython
Marcvs
Marcvs

Reputation: 125

PySide/Cython and GIL multi-thread use

I've been struggling for days with a PySide/Cython multi-thread application.
Splitting problems into parts, I came into a test I run with valgrind/helgrind (see one error entry below).
The C function (CHLone load below) uses Python with its GIL (no directive to Cython), Shiboken also accesses to the GIL.
The output suggest there is a conflict with GIL access (unless I'm wrong), but my understanding is that the GIL was there to avoid such a conflict.

Shiboken performs some dealloc that collides with the PyList_New on the other thread...

I'm lost,
I thought the GIL was in charge of blocking PyList_New while another thread was modifying some Python shared data. Or does Shiboken forget to lock ?

The listobject.c line 161 is a call to _PyObject_GC_TRACK() which (I assume) ask the GC to track the new object.
The classobject.c line 2360 is a call to _PyObject_GC_UNTRACK() which strongly looks like asking the GC to stop tracking an object...
I see the helgrind diag is a possible data race, but as a matter of fact this leads to core dump and I don't like that possible word on Python GC while using threads. I'd like to fix this one first.

==26535== ----------------------------------------------------------------
==26535== 
==26535== Possible data race during read of size 8 at 0x4FE8488 by thread #2
==26535== Locks held: none
==26535==    at 0x4C92A80: PyList_New (listobject.c:161)
==26535==    by 0x742C43F: s2p_parseAndReadHDF (SIDStoPython.c:949)
==26535==    by 0x742C5C4: s2p_parseAndReadHDF (SIDStoPython.c:968)
==26535==    by 0x742E638: s2p_loadAsHDF (SIDStoPython.c:1485)
==26535==    by 0x741C3CC: __pyx_f_6CHLone_load (pyCHLone.c:2182)
==26535==    by 0x741D2AD: __pyx_pf_6CHLone_12load (pyCHLone.c:2422)
==26535==    by 0x741D1C3: __pyx_pw_6CHLone_13load (pyCHLone.c:2392)
==26535==    by 0x4D0A48F: PyEval_EvalFrameEx (ceval.c:4013)
==26535==    by 0x4D0C3DC: PyEval_EvalCodeEx (ceval.c:3253)
==26535==    by 0x4C8B641: function_call (funcobject.c:526)
==26535==    by 0x4C5F652: PyObject_Call (abstract.c:2529)
==26535==    by 0x4C7279E: instancemethod_call (classobject.c:2578)
==26535== 
==26535== This conflicts with a previous write of size 8 by thread #1
==26535== Locks held: none
==26535==    at 0x4C6C53F: instancemethod_dealloc (classobject.c:2360)
==26535==    by 0x5AB248A: Shiboken::AutoDecRef::~AutoDecRef() (in /tmp/tools-2/local/x86z/lib/python2.7/site-packages/PySide/QtCore.so)
==26535==    by 0x5F9736F: PySide::GlobalReceiverV2::qt_metacall(QMetaObject::Call, int, void**) (in /tmp/tools-2/local/x86z/lib/libpyside-python2.7.so.1.0.9)
==26535==    by 0x659BCA5: QObject::event(QEvent*) (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==    by 0x5B038C5: QCoreApplicationWrapper::notify(QObject*, QEvent*) (in /tmp/tools-2/local/x86z/lib/python2.7/site-packages/PySide/QtCore.so)
==26535==    by 0x6586F8B: QCoreApplication::notifyInternal(QObject*, QEvent*) (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==    by 0x658A5A7: QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==    by 0x65B60F2: ??? (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==

Questions:
1. Is the behavior I assume for the GIL is correct?
2. If No,
... Does it mean I have to manage the GIL lock in each thread by myself?
... Then I have to create my own mutex for Python?!??!?
3. If Yes,
... Is there a problem in my Cython or C lib, regarding the GIL use?
... Could it be a QThread/QMutex side effect?

Upvotes: 3

Views: 1983

Answers (1)

user1969453
user1969453

Reputation:

In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once.

So all you can expect from the GIL is that it keeps two streams of python code from executing at once. You should not expect any sort of locking behavior beyond that. It is generally bad practice to write code that depends on the GIL protecting you. The GIL is a wart that many want to remove and it does not exist in some forms of python like IronPython and jython. Where it does exist it reduces concurrency, imposes overhead and makes python code harder to reason about than it would otherwise be.

It is not clear to me from the above that there is any GIL failure involved. The second trace seems to be deep in c++ land and not in areas that I understand that the GIL restricts.

Upvotes: 3

Related Questions