CODEWITHSUNDEEP
pythoncpythonperfgil
Olumide
Olumide

Reputation: 5829

CPython: what happens when the take_gil function calls the drop_gil function

I'm using perf probes to profile the GIL contention in a multithreaded Python application and I find sequences where the take_gil function calls drop_gil as shown in the following perf script dump of the perf data captured:

       viztracer  5533 [003]  3220.317244274:         python:take_gil: (55fe99b0d01e)
       viztracer  5533 [003]  3220.317407813:         python:drop_gil: (55fe99b0cf20)
       viztracer  5533 [003]  3220.317412443: python:drop_gil__return: (55fe99b0cf20 <- 55fe99baaa9e)
       viztracer  5533 [003]  3220.317419189:         python:take_gil: (55fe99b0d01e)
       viztracer  5533 [003]  3220.317422951:         python:drop_gil: (55fe99b0cf20)
       viztracer  5533 [003]  3220.317425869: python:drop_gil__return: (55fe99b0cf20 <- 55fe99baaa9e)

The corresponding part of take_gil in the CPython code where this happens looks like:

if (_PyThreadState_MustExit(tstate)) {
    /* bpo-36475: If Py_Finalize() has been called and tstate is not
       the thread which called Py_Finalize(), exit immediately the
       thread.

       This code path can be reached by a daemon thread which was waiting
       in take_gil() while the main thread called
       wait_for_thread_shutdown() from Py_Finalize(). */
    MUTEX_UNLOCK(gil->mutex);
    /* tstate could be a dangling pointer, so don't pass it to
       drop_gil(). */
    drop_gil(interp, NULL, 1);
    PyThread_exit_thread();
}

My question is, underwhat condition is this block of code executed? Is the thread that is called about to terminate, as PyThread_exit_thread() would seem to indicate. The perf script dump however suggests that the thread PID 5533 immediately reattempts to take the GIL at timestamp 3220.317419189 after previously dropping it at timestamp 3220.317412443, therefore the thread PID 5533 was not terminated.

Upvotes: 0

Views: 110

Answers (1)

Dunes
Dunes

Reputation: 40833

This code block is solely about daemon threads. Daemon threads will not keep a process alive if there are no other alive non-daemon threads. See https://docs.python.org/3/library/threading.html#threading.Thread.daemon

Basically, the last non-daemon thread has exited, and python is in the process of shutting down. No thread is allowed to continue executing python code at this point. From this point on it is just cleanup and making sure the process does not hang. See this note on the docs:

Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.

That's what this code is about, the process is terminating, but a daemon thread has tried to acquire the GIL. This is being denied, and the thread is being terminated immediately by PyThread_exit_thread(). It's not nice and it's not clean, but it does ensure the process exits.

Upvotes: 0

Related Questions