How do I handle errors when overwriting an existing file in python

Question

When I try to write to a file that already exists, but the write fails, the file gets overwritten with nothing (i.e. it gets cleared - contents of the existing file are deleted).

Example:

alaa@vernal:~/Test$ echo "sometext" > testfile
alaa@vernal:~/Test$ cat testfile 
sometext

In Python:

with open('testfile', 'w') as f:
    f.write(doesntexist)

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
NameError: name 'doesntexist' is not defined

The file is empty:

alaa@vernal:~/Test$ cat testfile 
alaa@vernal:~/Test$

I tried to handle the exception like this but it still empties the file:

import traceback

with open('testfile', 'w') as f:
    try:
        f.write(doesntexist)
        print('OK')
    except:
        print('NOT OK')
        traceback.print_exc()

NOT OK
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
NameError: name 'doesntexist' is not defined

I tried handling it in a different way, but it still empties the file:

import traceback

try:
    with open('testfile', 'w') as f:
        f.write(doesntexist)
        print('OK')
except:
    print('NOT OK')
    traceback.print_exc()

NOT OK
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
NameError: name 'doesntexist' is not defined

I feel like I'm missing something. Is there a straight forward way of handling this situation in Python to not clear the file if the write fails for any reason? Or do I have to resort to 'cleverer' code, for example: read the contents of the file first, and if the write fails, write it back again with the original contents, or maybe attempt to write the data to a temp file first, etc.?

Answer 1

I don't have the reputation to comment on an existing post, and I'd rather not edit someone else's answer, so I'm just going to post this separate answer.

@chepner's answer is I think the right solution, but there's a couple of mistakes and opportunities for simplification:

• safe_write is missing the @contextmanager decorator.
• The inner try-else block is invalid, so it should be changed to one of the following:
  • try-except-else (in which case the except should just do a raise to re-raise the existing exception)
  • Consolidate the inner try block with the outer one. I believe there is no point to an else clause in a try block if you're not going to handle the exception, so the contents of the else block could just happen immediately after the with block.

Also, os.replace will clobber the destination on all POSIX and Windows, whereas os.rename will only do so on POSIX platforms.

Finally, and this is personal preference since I know in Python it's perfectly okay to rely on exceptions as a means of control flow, but I've tweaked the code to just use f as the indicator of whether the file was successfully renamed.

Going with the approach of consolidating the try blocks, the overall solution would be:

from tempfile import NamedTemporaryFile
from contextlib import contextmanager

@contextmanager
def safe_write(fname, dir=None):
    if dir is None:
        dir = os.path.dirname(fname)
    try:
        f = None
        with NamedTemporaryFile('w', 
                                dir=dir,
                                delete=False) as f:
            yield f
        os.replace(f.name, fname)
        f = None
    finally:
        if f is not None:
            os.remove(f.name)

with safe_write('testfile') as f:
    f.write(doesntexist)

Answer 2

The exception that you're handling happens already after the file has been open for writing and truncated.

open('testfile', 'w')

Does open the file for writing and truncates it. See docs:

mode is an optional string that specifies the mode in which the file is opened... 'w' for writing (truncating the file if it already exists)...

Then:

f.write(doesntexist)

raises NameError because you're trying to access a variable doesntexist which indeed doesn't exist. At this point (handling this exception) there isn't anything you could do to change behavior of open.

As to what to do, sort of depends on exact use case, there are other options.

For the simple example above (simply overwrite file with content of a str variable if it exists), you could do this for instance:

from pathlib import Path
try:
    Path("testfile").write_text(doesntexist)
    print("OK")
except NameError:
    print("Not OK")

This way the exception is raised when calling the method and before file gets opened and write is attempted.

You could open the file for appending with a mode, but then the writes are of course appended.

You can prepare content to write making sure you have all bits and only open and write when you know it's safe to do so.

...

---

In other words, it's not the write that fails... and write as such failing after the open succeeds is relatively less likely (you could run out of space on the device for instance ENOSPC - you'd see OSError).

The specific failure you're seeing is due to accessing non-existing variable, so it's not about what happens prior to your write code, you can also:

stuff_to_write = doesntexist
with open('testfile', 'w') as f:
    f.write(stuff_to_write)

Which would hit that NameError before you open and truncate.

Or:

try:
    stuff_to_write = doesntexist
    with open('testfile', 'w') as f:
        f.write(stuff_to_write)
except Exception:
    print('Oops')

Even though that's a little silly.

Actually the specific case of variable not existing can either happen by simple omission (and any linter or check or just running the script should discover that) or the variable only being assigned to inside a conditional block which I would consider the correct place to address that problem.

Answer 3

open(..., 'w') immediately truncates the file, before you even have a chance to see if the subsequent writes succeeds.

Assuming you aren't required to modify the existing file in-place, it is far simpler and safer to create a new file that you will rename once the write has succeeded. You can wrap this in a context manager to simplify your business logic.

from tempfile import NamedTemporaryFile
from contextlib import contextmanager


def safe_write(fname, dir=None):
       
    if dir is None:
        dir = os.path.dirname(fname)
    try:
        try:
            f = None
            with NamedTemporaryFile('w', 
                                    dir=dir,
                                    delete=False) as f:
                yield f
        else:
            os.rename(f.name, fname)
    finally:
        try:
            if f is not None:
                os.remove(f.name)
        except FileNotFoundError:
            pass  # It doesn't exist if we successfully renamed it.


with safe_write('testfile') as f:
    f.write(doesntexist)

We create the temporary file in the same directory as the original to ensure they live on the same file system (but allow an explicit directory to be named if using the target directory isn't appropriate). delete=False ensures we still have a file to rename after we are done with the file.

The only thing this code tries to do with the original file is replace it atomically with the temporary file: the rename either succeeds, or leaves you with the original file still intact. Atomic renames are only possible within a file system, though. Moving a file to another file system requires actually copying the data in the file, rather than just updating its file-system entry.

---

Safely editing a file in-place is trickier and not something I'll attempt to discuss here.