keep having permission error while creating a hdf5 file

Question

I have the following segment of code to create a hdf5 file, and have used "with" statement to ensure the file is correctly closed. However, I still keep having the error message as follows.

   filename = 'E30.hdf5'
   try:
        with h5py.File(filename, 'w-') as f:
            print('---')
    except: 
        os.remove(filename)
        f = h5py.File(filename, 'w-')     

However, I still keep having the error message as follows. In the working directory, there may already have an existing file with the name of 'E30.hdf5'. But does it really matter? I tried to delete it from windows directly. However, the windows does not allow me to delete it saying it is being opened.

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-6-e8ccfbc1b5d2> in vid_to_hdf(En, start, end, chunk)
      9     try:
---> 10         with h5py.File(filename, 'w-') as f:
     11             print('---')

~\AppData\Local\Continuum\anaconda3\envs\fastai-py37\lib\site-packages\h5py\_hl\files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds)
    407                                fapl, fcpl=make_fcpl(track_order=track_order),
--> 408                                swmr=swmr)
    409 

~\AppData\Local\Continuum\anaconda3\envs\fastai-py37\lib\site-packages\h5py\_hl\files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    176     elif mode in ['w-', 'x']:
--> 177         fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
    178     elif mode == 'w':

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\h5f.pyx in h5py.h5f.create()

OSError: Unable to create file (unable to open file: name = 'E30.hdf5', errno = 17, error message = 'File exists', flags = 15, o_flags = 502)

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
<timed eval> in <module>

<ipython-input-6-e8ccfbc1b5d2> in vid_to_hdf(En, start, end, chunk)
     11             print('---')
     12     except:
---> 13         os.remove(filename)
     14         f = h5py.File(filename, 'w-')
     15     # Create dataset within file

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'E30.hdf5'

Answer 1

You are running into multiple issues at once. First, let's start with the h5py.File() access_mode flag.

• w- : Create file, fail if exists (avoids accidentally overwriting an existing file)
• w : Create file, truncate if exists (means it overwrites an existing file)
• r+ : Read/write, file must exist (use to open an existing file to write data).

In your logic below, your try:/except: pattern will execute the try: statement if E30.hdf5 does not exist. It will execute the except: statement if E30.hdf5 exists.

This is complicated by different h5py.File() methods is each branch. Your try: branch uses the with h5py.File() as f: method. So, when your code executes this logic, the file will close cleanly at the end (without a f.close() statement).

HOWEVER, your except: branch uses f=h5py.File(). So, when your code executes this logic, you need a f.close() statement to ensure closure at the end.

This is the scenario I think you are experiencing:

1. I assume E30.hdf5 does not exist the first time you run your code.
2. So, the first time you run, you go through the try: branch and the file is closed cleanly at the end.
3. The next time you run the code, E30.hdf5 exists, so, you go through the except: branch. As a result, the file is NOT closed at the end of the process, and another process cannot access it (Python or the OS).

Coding suggestions:

Your except: block has the same behavior mode=w. The code below behaves the same, and will always close the file when the process completes. Also, it is more readable (IMHO). Note: both methods delete E30.hdf5 if it exists.

filename = 'E30.hdf5'
with h5py.File(filename, 'w') as f: # use mode=w
     print('---')

Make this change if there is a burning need to keep the try:/except: pattern: (it's useful to use try:/except: for access modes w- and r+ WITHOUT the os.remove(filename).)

filename = 'E30.hdf5'
try:
    with h5py.File(filename, 'w-') as f:
        print('---')
except: 
    os.remove(filename)
    with h5py.File(filename, 'w-') as f:
         print('+++')