CODEWITHSUNDEEP
pythonmultithreadinggil
Nicolas S.Xu
Nicolas S.Xu

Reputation: 14534

Will Python Daemon thread be blocked by busy parent thread given there is GIL?

I know Python daemon thread will auto exit if parent thread exit. The reason I ask if because my main thread is sometime working on CPU heavy calculation which blocks the heart beat message. Other module thinks this service is dead and planning for funeral.

I want to use some kind of daemon thread(or process?) that is the child of main thread, to send heart beat message.

My question is if my main thread is blocking, for example:

while True:
   a = a + 1

Will my daemon child thread be blocked as well? any example or evidence? What role does GIL (Global Interpreter Lock) play here?

Edit

To @ShadowRanger's point, the following example shows, waiter can log out the the message, given the worker thread is blocking (taking 220% CPU resource)

import time
from threading import Thread

class worker(Thread):
    def run(self):
        t = 10
        x = 10
        while True:
            x = t + 1

class waiter(Thread):
    def run(self):
        for x in xrange(100,150):
            print x
            time.sleep(0.5)

def run():
    worker().start()
    waiter().start()

run()

Upvotes: 0

Views: 619

Answers (3)

Nicolas S.Xu
Nicolas S.Xu

Reputation: 14534

It is probably the IO loop callback inside a thread that blocks the heart beat. As @ShadowRanger points out, the thread is interleaving, which should not block another thread periodical callback (except the problematic python extension). In my case, it is probably the ioloop callback queue inside the thread is the culprit. No interleaving in ioloop callback queue.

Upvotes: 0

Tupteq
Tupteq

Reputation: 3095

For safety reasons I'd go with multiprocessing here. Your heartbeat response could be a separate process released from chains of GIL. Otherwise you'll depend on indeterministic code flow and you may get random errors in least expected moments.

Upvotes: 1

ShadowRanger
ShadowRanger

Reputation: 155428

If your main thread is executing Python byte code (not embedded in some expensive long term call to a C extension module that holds the GIL for the length of the call and doesn't return to the interpreter for a long time), then the daemon thread will eventually get a chance to run; the main thread will be preempted after a while (a matter of milliseconds) and the daemon thread will run while the main thread is blocked. As long as your heartbeat interval is measured in seconds, that should be fine.

Only one thread actually runs at any given time, but both will interleave execution.

The scenario with long term GIL holding by an extension can come up by accident, to be clear; all extensions default to holding the GIL. While popular, widely-used extensions like numpy are careful about releasing the GIL when they've got enough work to do, extensions written by less experienced developers (especially those writing in Cython with no knowledge of the GIL) are more likely to hold the GIL for long periods, so if your heartbeat isn't working and extension modules are involved, they're a likely culprit. A simple while True: a = a + 1 loop would be safe though (assuming a isn't some weirdo extension type); the GIL can be switched easily there.

Upvotes: 3

Related Questions