Registering a method as callback prevents the object from being garbage collected even with a finalizer that unregisters the callback

Question

The class in the real application is responsible for registering some of its methods (which internally use the file writer member) as callbacks to handle MQTT communication and for instantiating a new log writer (essentially a file stream wrapper). The file writer has a finalizer that closes the file, the main class has a finalizer that unregisters the callbacks (it also disconnects from the paho MQTT client and unsubscribes from the topics, but it is not relevant).

I understand the finalizer of the log writer cannot be called until the finalizer of the main class gets called because the methods registered as callbacks are still referencing the log writer. What I don't understand is why the finalizers don't get called at all.

I have tried to reproduce the issue with a dummy example. Here CallbackContainer is just a way to preserve the callable outside of the main Supervisor class.

import os
from weakref import finalize
from collections.abc import Callable
from typing import Optional


class FileStreamWrapper:

    def __init__(self, file: str):
        if os.path.isfile(file):
            os.remove(file)
        file_stream = open(file, "w")

        def finalizer():
            file_stream.close()
            print("finalized 'FileStreamWrapper' object")

        finalize(self, finalizer)
        self._file_stream = file_stream

    def print_line(self, line: str):
        self._file_stream.write(line + "\n")


class CallbackContainer:

    def __init__(self):
        self._callback: Optional[Callable[[], None]] = None

    def register_callback(self, callback: Callable[[], None]):
        self._callback = callback

    def unregister_callback(self):
        self._callback = None


class Supervisor:

    def __init__(self):
        file_stream_wrapper = FileStreamWrapper("test.txt")
        callback_container = CallbackContainer()
        callback_container.register_callback(lambda : self._print_line(""))

        def finalizer():
            callback_container.unregister_callback()
            print("finalized 'Supervisor' object")

        finalize(self, finalizer)
        self._file_stream_wrapper = file_stream_wrapper
        self._callback_container = callback_container

    def _print_line(self, line: str):
        self._file_stream_wrapper.print_line(line)


if __name__ == '__main__':

    for _ in range(2):
        supervisor = Supervisor()
        del supervisor

I expected both finalizers to be executed at end of the each iteration, after supervisor gets deleted. On the contrary, because the file is not closed, when creating a second instance of Supervisor a PermissionError exception is raised, forcing the interpreter to exit, which results in the previous finalizers being finally executed.

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'test.txt'
finalized 'FileStreamWrapper' object
finalized 'Supervisor' object

Answer 1

The callback registration is indeed blocking garbage collection of supervisor, not because the function keeps a strong reference to the file stream wrapper (and then the file stream) but because the function is also a method of the class. The issue might be overcome by registering a simple function in place of the method.

class Supervisor:

    def __init__(self):
        file_stream_wrapper = FileStreamWrapper("test.txt")
        callback_container = CallbackContainer()

        def print_line(line: str):
            file_stream_wrapper.print_line(line)

        callback_container.register_callback(lambda : print_line(""))

        def finalizer():
            callback_container.unregister_callback()
            print("finalized 'Supervisor' object")

        finalize(self, finalizer)
        self._file_stream_wrapper = file_stream_wrapper
        self._callback_container = callback_container

This indeed gives

finalized 'Supervisor' object
finalized 'FileStreamWrapper' object
finalized 'Supervisor' object
finalized 'FileStreamWrapper' object