Writing to file without losing data after crash

Question

I have a file that I open before a loop starts, and I'm writing to that file almost at each iteration of the loop. Then I close the file once the loop has finished. So e.g. something like:

testfile = open('datagathered','w')
for i in range(n):
    ...
    testfile.write(line)

testfile.close()

• The issue I'm having is that, in case the program crashes or I want to crash it, what has already been written to testfile will be deleted, and the text file datagathered will be empty. I understand that this happens because I'm closing the file only after the loop, but if I close and open the file after each write (i.e. in the loop) doesn't that lead to an incredible slow-down?

• If yes, what alternatives do I have for doing the writing, and making sure that in case of a crash the already-written-lines won't get lost, in an efficient way?

• The linked posts do bring up good suggestions that arguably answer this question, but they don't cover risks and efficiency differences involved. More precisely: Are there any risks involved with playing with the buffersize? e.g. testfile = open('datagathered','w',0) Finally is using with open... still a viable alternative if there are multiple files to write to?

• Small note: This is asked in the context of a very long run, where the file is being written to for 2-3 days. Thus having a speedy and safe way of doing the writing is definitely valuable here.

Answer 1

From the question I understood that you are talking about exceptions may occur at runtime and SIGINT.

You may use 'try-except-finally' block to achieve your goal. It enables you to catch both exceptions and SIGINT signal. Since the finally block will be executed either exception is caught or everything goes well, closing file there is the best choice. Following sample code would solve your problem I guess.

testfile = open('datagathered','w')
try:
     for i in range(n):
         ...
         testfile.write(line)
except KeyboardInterrupt:
    print "Interrupt from keyboard"
except:
    print "Other exception"
finally:
    testfile.close()

Answer 2

Use a context:

with open('datagathered','w') as f:
    f.write(data)