Attempt to open h5py file, returns errorno = 17, error message = 'file exists'

Question

import numpy as np import h5py

with h5py.File("testfile.hdf5", "w-") as f: arr = np.ones((5,2)) f["my dataset"] = arr dset = f["my dataset"]

This code runs correctly the first time, but when run a second time, returns the following error:

%run "C:\Users\James\Google Drive\Python Scripts\Python and HDF5\Chapter3.py" --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) C:\Users\James\Google Drive\Python Scripts\Python and HDF5\Chapter3.py in () 6 with h5py.File("testfile.hdf5") as f: 7 arr = np.ones((5,2)) ----> 8 f["my dataset"] = arr 9 dset = f["my dataset"] 10

h5py_objects.pyx in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py_objects.c:2696)()

h5py_objects.pyx in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py_objects.c:2654)()

C:\Users\James\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py_hl\group.py in setitem(self, name, obj) 291 else: 292 ds = self.create_dataset(None, data=obj, dtype=base.guess_dtype(obj)) --> 293 h5o.link(ds.id, self.id, name, lcpl=lcpl) 294 295 @with_phil

h5py_objects.pyx in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py_objects.c:2696)()

h5py_objects.pyx in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py_objects.c:2654)()

h5py\h5o.pyx in h5py.h5o.link (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py\h5o.c:3610)()

RuntimeError: Unable to create link (Name already exists)

%run "C:\Users\James\Google Drive\Python Scripts\Python and HDF5\Chapter3.py" --------------------------------------------------------------------------- IOError Traceback (most recent call last) C:\Users\James\Google Drive\Python Scripts\Python and HDF5\Chapter3.py in () 4 from timeit import timeit 5 ----> 6 with h5py.File("testfile.hdf5", "w-") as f: 7 arr = np.ones((5,2)) 8 f["my dataset"] = arr

C:\Users\James\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py_hl\files.py in init(self, name, mode, driver, libver, userblock_size, swmr, **kwds) 270 271 fapl = make_fapl(driver, libver, **kwds) --> 272 fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr) 273 274 if swmr_support:

C:\Users\James\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py_hl\files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr) 94 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl) 95 elif mode in ['w-', 'x']: ---> 96 fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl) 97 elif mode == 'w': 98 fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)

h5py_objects.pyx in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py_objects.c:2696)()

h5py_objects.pyx in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py_objects.c:2654)()

h5py\h5f.pyx in h5py.h5f.create (C:\pisi\tmp\h5py-2.6.0-2\work\h5py-2.6.0\h5py\h5f.c:2109)()

IOError: Unable to create file (Unable to open file: name = 'testfile.hdf5', errno = 17, error message = 'file exists', flags = 15, o_flags = 502)

The code and error were run in Canopy // Python 3.5. I also ran it in Spyder and received the same result. I also tried using

with h5py.File("testfile.hdf5", "a") as f:

with no success.

Answer 1

I encountered the exact same error message when I use HDF5Matrix class in keras(v2.2.2). However, I failed to find a mature solution to completely avoid this error when I have multiple training processes which all need to access the same HDF5 data on the disk. Only one process could successfully access this HDF5 data, while all others would report the same error, even though I modified the reading mode from the default r+ to r. I gave up and used the workable solution that keeps multiple copies of HDF5 data and one copy for each training process.

Answer 2

As per http://docs.h5py.org/en/latest/high/file.html, the w- mode is designed to cause the open operation to fail if the file already exists.