CODEWITHSUNDEEP
python
stenci
stenci

Reputation: 8481

Inconsistent "OSError: [Errno 22] Invalid argument" error

The add_to_log function defined in the following code is called thousands of times during the execution of a script:

LOGFILENAME = 'N:\\Tools\\DelOldTempSave.log'

def add_to_log(text):
    with open(LOGFILENAME, 'a') as logfile:
        now = str(datetime.datetime.now().replace(microsecond=0))
        logfile.write(now + ' ' + text + '\n')
        print(now + ' ' + text)

It usually works as expected, but once in a while it crashes with:

OSError: [Errno 22] Invalid argument: 'N:\\Tools\\DelOldTempSave.log'

I tested it 3 times, it crashed after 3200, 1700 and 10200 calls.

I think I can use a try / except block and give it another try when it fails, maybe after a time.sleep(0.1), but it's a little scary!

The open function is used also on other parts of the script. Does it mean that I need to use a try / except block wherever it is called?

Also, the open function is used on other scripts. Does it mean that I need to use a try / except block wherever it is called on whatever script?

And I can't trust python when writing to a text file?

EDIT

This version works without errors:

import os, time, datetime

LOGFILENAME = 'N:\\Tools\\text.log'
NROWS = 30000


def add_to_log(text):
    try:
        with open(LOGFILENAME, 'a') as logfile:
            now = str(datetime.datetime.now().replace(microsecond=0))
            logfile.write(now + ' ' + text + '\n')
            print(now + ' ' + text)
        time.sleep(0.1)
    except OSError as e:
        time.sleep(0.01)
        add_to_log(f'Crash: {e}')
        add_to_log(text)


if os.path.exists(LOGFILENAME):
    os.unlink(LOGFILENAME)

for n in range(NROWS):
    add_to_log(str(n))

There are still three problems with this:

  1. Decreasing the duration in time.sleep(0.1) increases the chances of error, so I'm not 100% sure 0.1 is a long enough duration to ensure no error
  2. How do I know that this problem is restricted to this function and it will not spread to other functions that call open
  3. Adding a 0.1 second pause on a function that is called 30000 times, adds 3000 seconds to the duration of the script

Note: N: is a network drive. I want the log file to be visible to all computers.

Upvotes: 0

Views: 304

Answers (1)

AKX
AKX

Reputation: 168913

And I can't trust python when writing to a text file?

Yes, it is a network drive. I want the log file to be visible to all computers.

It sounds more like you can't trust your network file system. Reliably appending to a file would require a (distributed network) lock to be taken out on the file, anyway, and if that works, it could be slow.

Rather than opening and closing a single log file every time you need to write something, I'd recommend generating a single random log file name at the start of your program, and writing there:

import secrets

LOGFILENAME = f'N:/Tools/text.{secrets.token_urlsafe(8)}.log'
LOGFILE = open(LOGFILENAME, "w")  # will be closed when the program ends


def add_to_log(text):
    now = datetime.datetime.now().replace(microsecond=0).isoformat()
    LOGFILE.write(f'{now} {text}\n')
    LOGFILE.flush()  # if you want to ensure the text is written
    print(f'{now} {text}')

If you need a single file later on, you can just concatenate all of those files together and sort them.

Upvotes: 2

Related Questions