CODEWITHSUNDEEP
pythonpython-2.7with-statement
NenadK
NenadK

Reputation: 391

writing to file does not complete even when using python with statement

I am using a simple python with statement such as below to write to a log file.

with open(filename, 'a+') as f:
    do_stuff1()
    f.write('stuff1 complete. \n')

    do_stuff2()
    f.write('stuff2 complete. \n')

    do_stuff3()
    f.write('stuff3 complete. \n')

I am finding that my script fails intermittently at do_stuff2() however in the log file I do not find the line "stuff1 complete" as I would expect if the file was closed correctly as should happen when using with. The only reason I do know the script is failing at do_stuff2() without my log working is because this function calls an API that does its own logging and that other log file tells me that 2 has been executed even if it did not complete.

My question is what sort of error would have to occur inside the with statement that would not only stop execution but also prevent the file from being closed correctly?

Some additional information:

  1. The script is a scheduled task that runs late at night.
  2. I have never been able to reproduce by running the process interactively.
  3. The problem occurs once in every 2-3 nights.
  4. I do see errors in the Windows event logs that point to a dll file, .NET Framework and a 0xC0000005 error which is a memory violation. The API used by do_stuff2() does use this DLL which in turn uses the .NET Framework.

Obviously I am going to try to fix the problem itself but at this point my question is focused on what could happen inside the with (potentially some number of layers below my code) that could break its intended functionality of closing the file properly regardless of whether the content of the with is executed successfully.

Upvotes: 2

Views: 649

Answers (1)

Mike Müller
Mike Müller

Reputation: 85512

The with can only close the file if there is an exception. If there is a segfault inside an extension, no exception might be raised and the process dies without giving Python a chance to close the file. You can try to use f.flush() at several places to force Python to write to the file.

Upvotes: 2

Related Questions