Python Doesn't Open File Even Though it Exists and Is in Correct Directory

Question

I have a Python script that opens a data file, finds the relevant lines, and stores the results as a list of tuples. In the script there is a function ValuesFromFile(year,month,day,soft_address) that accepts the string values year, month, day, and an integer soft_address which is to be found in the line. The function is used over a range or dates to look through daily files, located in another directory. In detail, the function,

def ValuesFromFile(year,month,day,soft_address):
    """Given the year, month, day, find all the lines that contain the soft address
       and call GetValues() on the line to get the associated values over the hour.

       The function builds the filename, checks if it exists, then goes line-by-line
       finding the lines with the soft address.

       Return a list of tuples [(datetime, value)]"""

    filename = '../../BEMS/' + str(year) + str(month).zfill(2) + '/' + 'gd' + str(year) + str(month).zfill(2) + str(day).zfill(2) + '.txt'
    data_list = []
    print filename
    try:
        FileObject = open(filename,'r')

        for line in FileObject:
            check_line = line.rsplit()
            hour_check = 0 # Check the hour in case we are missing an hour
            if (str(soft_address) in check_line):
                temp_list = GetValues(line)
                for tuple_value in temp_list:
                    date = "%s-%s-%s %s:%s"%(str(year),str(month),str(day),str(tuple_value[0]),str(tuple_value[1]))
                    data_list.append((datetime.strptime(date,"%Y-%m-%d %H:%M"),tuple_value[2]))
        FileObject.close()
    except:
        data_list = MissingDay(str(year),str(month),str(day))
        print "Couldn't find file"

    finally:
        FileObject.close()

In the above function I create the filename string by concatenating the location of the file, ../../BEMS/str(year)+str(month).zfill(2)+'/', then the actual name of the file, 'gd' + str(year) + str(month).zfill(2) + str(day).zfill(2) + '.txt'. If the file is found, I check if the sting soft address is in the line, and then call another function. If the file is not found, it will print "Couldn't find file."

When I run the script over the time span of 2017-10-01 to 2017-10-15, the file name is printed and then try-except-finally is executed. If the file is found and processed correctly, there is no output. If the file is not found the except is executed and the print statement printed. The results:

So the weird thing is that the script is able to find all of the other files. The file exists in the directory which is confirmed by checking: . Also, we can see that all users have read permissions on the file from the previous screenshot. I tried doing a chmod on the file just to see and that didn't help, as expected.

To add to the weirdness, when I try and read the file from the interactive interpreter, I am able to open and read the file:

Finally, I renamed the file to check.txt and put it in the same directory as the code, commented out the filename building, and replaced it with filename = "check.txt", reran the script, and the file still wasn't found by the script. So this seems like it is specific to that file.

Answer 1

As Kevin points out in the comments, you shouldn't use bare except blocks that assume a particular error occurred. Given the bug I pointed out in my comment (if open fails, the finally block will raise UnboundLocalError when it tries to call FileObject.close()), unless you're catching and suppressing the UnboundLocalError, the one thing that definitely isn't happening is a failure to open the file. More likely, your parsing is failing, and since you're treating all exceptions the same, you report a failure to parse as a failure to open the file.

To fix this, reorganize your code to avoid misreporting errors (and while you're at it, avoid closeing stuff that might not exist):

def ValuesFromFile(year,month,day,soft_address):
    """Given the year, month, day, find all the lines that contain the soft address
       and call GetValues() on the line to get the associated values over the hour.

       The function builds the filename, checks if it exists, then goes line-by-line
       finding the lines with the soft address.

       Return a list of tuples [(datetime, value)]"""

    # Simplify path construction just because
    filename = '../../BEMS/{y}{m:02d}/{y}{m:02d}{d:02d}.txt'.format(y=year, m=month, d=day)
    data_list = []    
    print filename
    try:
        FileObject = open(filename,'r')
    except EnvironmentError as e:
        print "Error opening file:", e
        return MissingDay(str(year),str(month),str(day))

    # No code should go here, as exceptions raised here don't benefit from with
    # statement guaranteed closure

    # with statement guarantees it's closed on block exit, whether or not exception occurs
    # Normally, you'd open it in the with statement too, but in this case,
    # you wanted to handle errors from open immediately, so we separate it a bit         
    with FileObject:
        for line in FileObject:
            check_line = line.rsplit()
            hour_check = 0 # Check the hour in case we are missing an hour
            if str(soft_address) in check_line:
                temp_list = GetValues(line)
                for tuple_value in temp_list:
                    date = "%s-%s-%s %s:%s"%(str(year),str(month),str(day),str(tuple_value[0]),str(tuple_value[1]))
                    data_list.append((datetime.strptime(date,"%Y-%m-%d %H:%M"),tuple_value[2]))

return data_list

This way you:

1. Only catch the error you intend to catch (EnvironmentError, which covers IOError and OSError on Py2, and aliases OSError from 3.3 onwards) from the one place you expect it (the open call)
2. Have an error message with more details, rather than blindly claiming it didn't find the file
3. All other exceptions from the rest of the code bubble uninterrupted, so you get a full stack trace and error message telling you what went wrong in the error cases you didn't expect