CODEWITHSUNDEEP
pythonmultithreading
The_Fallen
The_Fallen

Reputation: 327

Hanging thread in Python

I've got a somewhat larger Python project that's running multiple threads, some of which I start myself and others being started from external packages. Now in the case of one specific configuration, the program doesn't finish properly, there's one thread still running that blocks it. I cannot find any significant difference for this specific configuration, so I had to dig deeper.

What I do at the end of the program now is something like this:

while threading.active_count() > 1:
    names = [t.name for t in threading.enumerate() if t != threading.current_thread()]
    log.info('Waiting for threads to close: ' + ','.join(names))
    time.sleep(2)

If more than the main thread is still running, this constantly prints the list of remaining threads, except main. The name of the one remaining thread is always ThreadPoolExecutor-0_0. Just as a side note: I name all my threads, so I know for sure that's none of mine -- but of course the name suggests that anyway.

Now I tried to dig even deeper and print the stack trace of this remaining thread:

frame = sys._current_frames().get(thread.ident, None)
traceback.print_stack(frame)

Which gives me:

  File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
    work_item = work_queue.get(block=True)

Unfortunately this isn't much help, since, as expected, this just points to a worker method for a thread pool.

When looking at that method (_worker), I find this comment:

# Workers are created as daemon threads. This is done to allow the interpreter
# to exit when there are still idle threads in a ThreadPoolExecutor's thread
# pool (i.e. shutdown() was not called).

So, as I understand, this should not happen at all.

Any idea, what's going on here? How can I find out, where this thread had been started and why it's not closing properly?

Upvotes: 0

Views: 1675

Answers (1)

Lenormju
Lenormju

Reputation: 4378

Searching for ThreadPoolExecutor on a search engine give me the Python library concurrent.futures as first result.

It lists an Executor.shutdown method that :

Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. [...]

It looks like one of your dependency is using futures, which are executed in a thread pool handled by an executor that is never shutdown. The thread that is in the executor's pool is thus never killed.

Maybe you don't use one of your external libraries "the proper way", which causes the executor to outlast its scope (for example using a context manager), or maybe there is a shutdown/teardown/stop/exit/finish function that this library exposes to do it that you did not call.

The documentation states rightly for shutdown :

You can avoid having to call this method explicitly if you use the with statement, which will shutdown the Executor (waiting as if Executor.shutdown() were called with wait set to True)

But it is also possible that the external library just is not perfect or frankly misbehaves.
In that case, if you know that your program has really finished, you can assume that nothing is running in this thread and find the executor (in one of your external packages) and ask it yourself to shutdown.
You could also try to terminate the thread (but it is not a good practice to do so to threads you don't own).
Or just ignore this thread too and exit.

Upvotes: 1

Related Questions