CODEWITHSUNDEEP
pythonpython-3.xnumpyh5py
ojunk
ojunk

Reputation: 899

Python: h5py gives OSError: Can't read data (inflate() failed) even though it's opened it before

Python 3.5. I have a few hundred .mat mat files (version 7.3) in a directory. I am looping through all of them to extract two different parts of data. I loop through and get and get the first lot with no problems at all but when I do the exact same thing again only I extract a different part of the data I get the following error:

Traceback (most recent call last):
  File "v73_test.py", line 43, in <module>
    mrfs_data = extract.convert1simProteinComCountsIntoDataFrame(path2mats)
  File "/home/oli/Downloads/PhD/wc/mg/version_73_stuff/functions_for_joshuas_matFiles/extract_matFile_data_v73.py", line 586, in convert1simProteinComCountsIntoDataFrame
    raw_data = getMatureProteinComplexs(path2mats, state_no)
  File "/home/oli/Downloads/PhD/wc/mg/version_73_stuff/functions_for_joshuas_matFiles/extract_matFile_data_v73.py", line 53, in getMatureProteinComplexs
    if len(np.array(state_file['ProteinComplex']['counts']).shape) == 3:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/oli/virtualenvs/standard_python3.5/lib/python3.5/site-packages/h5py/_hl/dataset.py", line 696, in __array__
    self.read_direct(arr)
  File "/home/oli/virtualenvs/standard_python3.5/lib/python3.5/site-packages/h5py/_hl/dataset.py", line 657, in read_direct
    self.id.read(mspace, fspace, dest, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (inflate() failed)

So the file is definitely there and accessible so the only thing I can think of is the data is corrupted but if that is the case surely I wouldn't have been able to extract any data from it all?

Upvotes: 7

Views: 10085

Answers (2)

DarylVanDam
DarylVanDam

Reputation: 21

Python 3.6. I experienced a similar problem, whereby I could read output data (for a given grid cell in a 2D SHETRAN hydrological model) e.g., up to the 24th time step:

 f['VARIABLES']['  6 srf_dep']['value'][0,0,24]

 Out[272]: -1.0

But by the 240th time step, I was getting the same error:

['VARIABLES']['  6 srf_dep']['value'][0,0,240]


 Traceback (most recent call last):

  File "<ipython-input-273-1f12011309c3>", line 1, in <module>
    f['VARIABLES']['  6 srf_dep']['value'][0,0,240]

  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

  File "C:\Anaconda3\lib\site-packages\h5py\_hl\dataset.py", line 496, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)

  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

  File "h5py\h5d.pyx", line 181, in h5py.h5d.DatasetID.read

  File "h5py\_proxy.pyx", line 130, in h5py._proxy.dset_rw

  File "h5py\_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread

 OSError: Can't read data (inflate() failed)

I solved this by deleting the problematic H5 file, reopening my IDE and running again. Thanks for the solution.

Upvotes: 2

ojunk
ojunk

Reputation: 899

I answered my own question here because there is isn't much on the net about this error and I learnt something so maybe it will help someone else.

So I have realised that the data is in fact corrupted. I thought that if a file was corrupted then you would be able to extract nothing from it but it turns out that in this case this is not true and the only bit you can't access is the specific bit that is corrupted. This was not what I expected based on passed experience with other versions of .mat files but now I think about what this version actually is it seems kind of obvious.

Upvotes: 11

Related Questions