CODEWITHSUNDEEP
pythonpython-3.xtemporary-files
Thomas Kühn
Thomas Kühn

Reputation: 9830

Deleting temporary files during script evaluation

I'm using a python package, cdo, which heavily relies on tempfile for storing intermediate results. The created temporary files are quite large and when running bigger calculations, I've run into the problem that the /tmp directory got filled up and the script failed with a disk full error (we are talking about 10s to 100s of GB). I've found a workaround to the problem by creating a local folder, say $HOME/tmp and then doing

import tempfile
tempfile.tempdir='$HOME/tmp'

before importing the cdo module. While this works for me, it is somewhat cumbersome if I want also others to use my scripts. Therefore I was wondering, whether there would be a more elegant way to solve the problem by, e.g., telling tmpfile periodically to clear out all temporary files (usually this is only done once the script finishes). From my side this would be possible, because I am running a long loop, which produces one named file each iteration and all the temporary files created during that iteration would be discardable afterwards.

Upvotes: 1

Views: 393

Answers (1)

hiro protagonist
hiro protagonist

Reputation: 46921

as the examples show: you could use tempfile in a context manager:

with tempfile.TemporaryFile() as fp:
    fp.write(b'Hello world!')
    fp.seek(0)
    fp.read()

that way they are removed when the context exits.

...do you have that much control over how cdo uses tempfiles?

Upvotes: 2

Related Questions