CODEWITHSUNDEEP
pythonmultithreadinglockingslowdown
mztulip
mztulip

Reputation: 1

python main thread slow down with started second while loop thread

I have problem with following code. I started new thread which have empty while loop. It causes that function psutil.net_connections() is executed slowly. I need to wait about 1 minute to get result.

import psutil
import time
import threading

pid = os.getpid()
proc = psutil.Process(pid)

exit = False

def MyThread1():
    while True:
        if exit == True:
            break
        # time.sleep(1)


t1 = threading.Thread(target=MyThread1, args=[])
t1.start()

connections = psutil.net_connections()

for con in connections:
    print(con.laddr.port)

exit = True
t1.join()

When I press Ctrl+c when code is working. It shows me that I interrupted code during lock.acquire when readlink function is executed.

  File "/home/mz/pyenv/lib/python3.11/site-packages/psutil/_pslinux.py", line 215, in readlink
    path = os.readlink(path)
           ^^^^^^^^^^^^^^^^^
KeyboardInterrupt
^CException ignored in: <module 'threading' from '/usr/lib/python3.11/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1590, in _shutdown
    lock.acquire()
KeyboardInterrupt:

Somehow when I added time.sleep in while loop it works perfectly. Then I started to read thread python documentation and there is no information how scheduler works. I read that it switches task context every 100 byte codes(How does python handle thread locking / context switching?). It was surprise for me, but anyway I still can not understand why it behaves in that way. Can you help me to understand this case? My python version is Python 3.11.9.

Upvotes: 0

Views: 89

Answers (2)

mztulip
mztulip

Reputation: 1

After watching videos about GIL I want to make short explanation why presented code works in very slow way.

Python threads are real system pthreads(I thought that scheduler is implemented in python previously). Anyway it does not allow to execute code on multi core systems, due to GIL. Thread to execute code must get GIL mutex. Other threads must wait to release. Therefore, there is no parallelism, threads execute code one after another, even on multi core system.(reason is that it was designed in 1990 when single core computers were used)

Situation when thread releases mutex are triggered by IO access, for example usage of sockets, filesystem functions, sleep. There is also second option, GIL is released by default every 5ms in python3. In python 2.7 it was done every 100 instructions.

In my code, one thread have empty while loop which is CPU bounded and main thread is IO bounded because it gets information about sockets. Theoretically with things that I mentioned, my code should work. Thread with while loop should release GIL every 5ms and main IO thread should execute. In reality this doesn't work in that way. This is associated with specific GIL lock implementation(IO thread at first releases GIL when it get GIL) what causes that mainly CPU bound thread is executed. Details are explained in mentioned videos.

Adding sleep inside while loop solves problem because sleeping thread can not immediately be executed by Linux scheduler,even when IO thread releases GIL firstly, It immediately gets GIL mutex(there is no other active CPU bound thread) and can execute their own code.

In any of my dreams and my imaginations I would not discover presented code relations without mentioned videos.

Upvotes: 0

Ahmed AEK
Ahmed AEK

Reputation: 18090

The way to wait for an event to happen in another thread in python is to wait on a threading.Event

import threading

def MyThread1(event: threading.Event):
    event.wait()
    # do some work

wait_event = threading.Event()
t1 = threading.Thread(target=MyThread1, args=[wait_event], daemon=True)
t1.start()

# do some work

wait_event.set() # signal thread to stop waiting.
t1.join()

Spinning in a while predicate: loop is wrong and should be considered a bug, it wastes cpu cycles and blocks other threads from running due to the GIL.

The GIL switches every almost 5 milliseconds, controllable with sys.setswitchinterval(interval), this video goes into detail about how python3 gil works https://m.youtube.com/watch?v=Obt-vMVdM8s it is very unfair for IO, but basically an empty loop will hog most of the interpreter time

There are equivalent asyncio.Event and multiprocessing.Event for event that are not in another thread.

Upvotes: 0

Related Questions