How to write to a file from various places in the code in a pythonic and performant way

Question

When writing to a file in python, you should typically use the using structure in order to have the file closed after writing, like so:

with open("myfile.txt", "a") as file1:
    file1.write("Hello \n")

But if I, during the execution of my script, wants to write to the same file from different places in the code, I might be tempted to encapsulate the above structure in some kind of method like so:

def write_to_file(self, string_to_write):
    with open(self.myfile_path, "w") as file1:
       file1.write(f"{string_to_write}\n")

The above would likely give me a pretty bad performance hit since the file is opened every time I call the method.

The alternative that I can see is opening the file early in the program and having a file.close() call in some finally clause somewhere and hope for the best. But I understand this to be associated with some risk.

So given the above, how should one approach this task in a pythonic as well as a performant way?

Answer 1

Unless you can put everything into with block like in scotscotmcc's answer, I'd probably just do it without with. Just open the file, write to it, then .close() it.

But here's an idea for using with and not putting the code into the with block. A generator that writes to the file what you send it:

def printer(filename):
    with open(filename, 'w') as f:
        while True:
            print((yield), file=f)

# Demo usage
p = printer('test.txt')
next(p)
p.send('foo')
p.send('bar')
p.close()

# Check the resulting file
with open('test.txt') as f:
    print(repr(f.read()))

Output (Try it online!):

'foo\nbar\n'

I don't think it's any better than just open+close a file, though. I just tried it for fun / out of curiosity.

Answer 2

Based on @CharlieBONS comment and some Singleton patterns for python, I created this solution. I still suspect there is something better, closer to the file loggers in logging, but I have not had the time to dive into that.

The previous solution was based on logging to a file with a specific logging level, but since the python library needs to hook into other solutions with their own logging configurations, like AirFlow, i had to abandon this.

import json
import logging
import os
from pathlib import Path
from typing import List


class MyWriter:

    __instance = None
    __inited = False

    def __new__(cls, path_to_file: Path) -> "MyWriter":
        if cls.__instance is None:
            cls.__instance = super().__new__(cls)
        return cls.__instance

    def __init__(self, path_to_file: Path) -> None:
        if type(self).__inited:
            return
        self.cache: List[str] = []
        self.path_to_file: Path = path_to_file
        if self.path_to_file.is_file():
            os.remove(self.path_to_file)
        type(self).__inited = True

    def write(self, string_to_write: str, flush=False):
        try:
            if string_to_write:
                self.cache.append(f"{string_to_write)}\n")
            if len(self.cache) > 1000 or flush:
                with open(self.path_to_file, "a") as my_file:
                    extradata_file.writelines(self.cache)
                    self.cache = []
                    logging.debug("My Writer flushing the cache")
        except Exception as ee:
            error_message = "Something went wrong in My Writer"
            logging.error(error_message)
            raise ee

    def flush(self):
        self.write("", True)

Answer 3

You can have your with statement early in your code and have all the other functions that use the file (and probably many that don't use the file but are called in between the ones that do) indented from it and pass the file to them.

This may not be wonderful to refactor things to this given the current code and complexity...

def main():
    with open('file.txt','w') as file:
        my_func_1()
        my_func_2(file)
        my_func_3
        my_func_4(file)
        ...

def my_func_1():
    ...

def my_func_2(file):
    ...
    file.write('thing to write')
    ...

def my_func_3():
    ...

def my_func_4(file):
    ...
    file.write('thing to write')
    ...