CODEWITHSUNDEEP
python-3.xpython-c-api
Matthew Moisen
Matthew Moisen

Reputation: 18299

Do Python C Extensions release the GIL for CPU tasks as they do for IO tasks?

Python C Extensions always release the GIL via Py_BEGIN_ALLOW_THREADS when making IO system calls. I have read mixed opinions on whether C Extensions release the GIL for relatively long running CPU-bound tasks. I've read that CPU inside the python interpreter is subject to GIL but that CPU inside of a C Extension often releases the GIL. I assume this would be safe and efficient to do so as long as the CPU tasks could be performed in a temporary variable with fine-grain locking, and the GIL was acquired before an update to a shared structure takes place.

I searched through the 3.6 C code in Github for all instanced of Py_BEGIN_ALLOW_THREADS. I might have missed some but it seems to me that Python C Extensions in general do not release the GIL for CPU bound tasks.

Do Python C Extensions typically release the GIL for CPU tasks as they do for IO tasks?

Here are the only examples I saw where they did release the GIL for a CPU task:

The _hashlib.h has a macro ENTER_HASHLIB with a comment indicating that it will release the GIL around a CPU consuming hashlib operation.

The sha3module.c appears to allow threads before a CPU operation.

The _lzmamodule.c appears to allow threads before a CPU operation.

I could not find any other CPU calls that released the GIL (I definitely might have missed some, let me know if I did).

For example:

The _bisectmoduel.c has no Py_BEGIN_ALLOW_THREADS.

The _heapqmodule.c has no Py_BEGIN_ALLOW_THREADS.

The _json.c has no Py_BEGIN_ALLOW_THREADS.

The _csv.c has no Py_BEGIN_ALLOW_THREADS.

Upvotes: 0

Views: 634

Answers (1)

Roland Smith
Roland Smith

Reputation: 43523

Well, numpy does it, although they wrap their own macro around it (NPY_BEGIN_ALLOW_THREADS / NPY_END_ALLOW_THREADS)

According to the docs:

This group is used to call code that may take some time but does not use any Python C-API calls. Thus, the GIL should be released during its calculation.

So you can release the GIL, but only if you do not use any Python C-API calls. Another way of saying that is if you do not access Python objects.

But the code in the last four extension modules you linked is filled with C-API calls. So there is probably not enough calculations done in C in those modules that do not involve C-API calls to make releasing and re-acquiring the GIL worthwhile.

Upvotes: 1

Related Questions