Proper method for terminating a python program regardless of call location, with cleanup

Question

How does one properly and cleanly terminate a python program if needed? sys.exit() does not reliably perform this function as it merely terminates the thread it is called from, exit() and quit() are not supposed to be used except in terminal windows, raise SystemExit has the same issues as sys.exit() and is bad practice, and os._exit() immediately kills everything and does not clean up, which can cause issues with residuals.

Is there a way to ALWAYS kill the program and all threads, regardless of where it is called from, while still cleaning up?

Answer 1

Is there a way to ALWAYS kill the program and all threads, regardless of where it is called from, while still cleaning up?

No - "regardless where it is called from" and "cleaning up" do not mix.

---

It is simply not meaningful to both reliably and safely kill a thread. Killing a thread (or process) means interrupting what it is doing - that includes clean up. Not interrupting any clean up means, well, not actually killing a thread. You cannot have both at the same time.

If you want to kill all threads, then os._exit() is precisely what you are asking for. If you want to clean up a thread, no generic feature can fulfil that.

The only reliable way to shut down threads is to implement your own, safe interrupt. To some extent, this must be customised to your use case - after all, you are the only one knowing when it is safe to shut down.

Killing a thread with Exceptions

The underlying CPython API allows you to raise an exception in another thread. See for example this answer.

This is not portable and not safe. You could be killing that thread at any arbitrary point. If your code expects an exception or your resources clean up after themselves (via __del__), you can limit harm, but not exclude it. Still, it is very close to what most people think of as a "clean kill".

Self-Cleaning daemon threads with atexit

Threads running with Thread.daemon are abruptly terminated if no other threads remain. Usually, this is half of what you want: gracefully terminate if all proper threads exit.

Now, the key is that a daemon thread does not prevent shutdown. This also means it does not prevent atexit from running! Thus, a daemon can use atexit to automatically shutdown itself and cleanup on termination.

import threading
import atexit
import time


class CleanThread(threading.Thread):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.daemon = True
        # use a signal to inform running code about shutdown
        self.shutdown = threading.Event()

    def atexit_abort(self):
        # signal to the thread to shutdown with the interpreter
        self.shutdown.set()
        # Thread.join in atexit causes interpreter shutdown
        # to be delayed until cleanup is done
        self.join()  # or just release resources and exit

    def run(self):
        atexit.register(self.atexit_abort)
        while not self.shutdown.wait(0.1):
            print('who wants to live forever?')
        print('not me!')
        atexit.unregister(self.atexit_abort)


thread = CleanThread()
thread.start()
time.sleep(0.3)
# program exits here

Note that this still requires your code to listen for a cleanup signal! Depending on what the Thread does, there are other mechanisms to achieve this. For example, the concurrent.future module empties the task queue of all worker threads on shutdown.

Answer 2

While your app is running Bad Things can happen, which we'd like to recover from. One example is Power Fail.

There is no computing technique for arranging for instructions to execute on a device that is powered off. So we may need to reset some state upon restart. Your app already has this requirement; I'm just making it explicit.

It is hard to reliably gain control just after each of the various Bad Things that might happen, as you found when you carefully considered several standard techniques. You weren't specific about the sort of items needing cleanup that you envision, but we could consider these cases:

1. transient - TCP connections, flock, etc.
2. permanent - disk files, side effects on distant hosts

Rather than invoking your app directly, arrange for it to be run by a Nanny process which forks the app as a child. At some point the app will exit, the Nanny will regain control with all transient items having been tidied up by the OS, and then the Nanny can do any necessary cleanup on permanent items prior to an app restart. This is identical to the cleanup the Nanny will need to do on initial startup, for example after power fail events. The advantage of running your app under a parent process is that the parent can perform immediate cleanups after simple app failures such as SEGV.

Cleaning up permanent items likely involves timeouts on timestamped resources. If your system is able to reboot within say 2 seconds of a brief power outage, you may find it necessary to deliberately stay Down (sleep) for long enough to ensure that distant hosts have reliably detected your transition to Down, prior to announcing a transition to Up. Techniques like Virtual Synchrony and Paxos can help you drive toward rapid convergence.

summary

Sometimes an app will unexpectedly die before running cleanup code. Take a belt-and-suspenders approach: put essential cleanup code in the (simpler, more reliable) parent process.