How to use context managers with callback functions?

Question

How do you correctly use resources, when the resource is expected to be released asynchronously?

Motivated by tempfile.TemporaryDirectory emitting a warning, when the temporary directory is cleaned up by the destructor rather than explicitly, and this leading to problems in PYTHONSTARTUP. Note that the parameter ignore_cleanup_errors does not affect the warning.

Toy example with lexical closures.

Consider e.g. the following nonsense example:

import tempfile, os

def make_writer():
    tmpdir = tempfile.TemporaryDirectory()
    counter = 0
    def writer():
        nonlocal counter
        counter += 1
        filename = os.path.join(tmpdir.name, f"{counter}.dat")
        with open(filename, "w") as fout:
            print(f"{counter = }", file=fout)
        os.system(f"tree '{tmpdir.name}'")
    return writer

writer = make_writer()
writer()
writer()

This will output e.g.

/tmp/tmp9ke4swfe
└── 1.dat

0 directories, 1 file
/tmp/tmp9ke4swfe
├── 1.dat
└── 2.dat

0 directories, 2 files
/usr/local/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp9ke4swfe'>
  _warnings.warn(warn_message, ResourceWarning)

Normally, I'd rewrite this to somehow use a context manager, but what, if writer is used as e.g. an asynchronous callback? Or a utility to be used in interactive python shells? In that case, using a context manager doesn't seem possible.

Toy example with a class.

So, next I tried to write a wrapper class, that explicitly cleans up the temporary directory, when it is destroyed. Remember, this is going into PYTHONSTARTUP, so we can't explicitly clean up the object.

import tempfile, os, sys

class Writer:
    def __init__(self):
        self.tempdir = tempfile.TemporaryDirectory()
        self.counter = 0
    def __call__(self):
        self.counter += 1
        filename = os.path.join(self.tempdir.name, f"{self.counter}.dat")
        with open(filename, "w") as fout:
            print(f"{self.counter = }", file=fout)
        os.system(f"tree '{self.tempdir.name}'")
    def __del__(self):
        sys.stdout.flush()
        sys.stderr.flush()
        print("Executed after the cleanup warning?")
        sys.stdout.flush()
        sys.stderr.flush()
        self.tempdir.cleanup()

writer = Writer()
writer()
writer()

Which generates:

/tmp/tmpmwrb2nhf
└── 1.dat

0 directories, 1 file
/tmp/tmpmwrb2nhf
├── 1.dat
└── 2.dat

0 directories, 2 files
/usr/local/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpmwrb2nhf'>
  _warnings.warn(warn_message, ResourceWarning)
Executed after the cleanup warning?

Note that the warning is already emitted, before the destructor of the manager class is called. That part I currently don't understand at all.

Toy example using generators

For completeness sake, this also doesn't solve the problem:

import tempfile, os

def make_writer():
    def make_writer_obj():
        with tempfile.TemporaryDirectory() as tempdir:
            counter = 0
            while True:
                counter += 1
                filename = os.path.join(tempdir, f"{counter}.dat")
                with open(filename, "w") as fout:
                    print(f"{counter = }", file=fout)
                os.system(f"tree '{tempdir}'")
                yield
    writer_obj = make_writer_obj()
    return lambda: next(writer_obj)

writer = make_writer()
writer()
writer()

Output:

/tmp/tmpa6t11t5e
└── 1.dat

0 directories, 1 file
/tmp/tmpa6t11t5e
├── 1.dat
└── 2.dat

0 directories, 2 files
/usr/local/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpa6t11t5e'>
  _warnings.warn(warn_message, ResourceWarning)

Current solution: atexit.register

The current use-case of customizing the interactive shell I have solved by using

tempdir = tempfile.TemporaryDirectory()
atexit.register(tempdir.cleanup)

The downside of this solution is, that it will potentially do the cleanup unnecessarily late for uses, where cleanup could be performed earlier.