CODEWITHSUNDEEP
pythonnumpysliceh5py
Steve
Steve

Reputation: 375

Use python slice objects when reading HDF5 file with h5py?

I am trying to use python slice objects to access data from an HDF5 file using the h5py module. I put together this example to show that it works with numpy arrays, but not with h5py.

import h5py
import numpy as np

slice_obj = [slice(None,3,None), slice(2,5,None)]

test_array = np.ones((3,5))
print test_array[0:3,2:5]
print test_array[slice_obj]

f = h5py.File("testing.hdf5","w")
f['data'] = test_array
f.close()

f = h5py.File("testing.hdf5","r")
test2 = f['data'][0:3,2:5]
print test2
test2 = f['data'][slice_obj]
print test2
f.close()

This gives the following output:

[[ 1.  1.  1.]
 [ 1.  1.  1.]
 [ 1.  1.  1.]]
[[ 1.  1.  1.]
 [ 1.  1.  1.]
 [ 1.  1.  1.]]
[[ 1.  1.  1.]
 [ 1.  1.  1.]
 [ 1.  1.  1.]]
Traceback (most recent call last):
  File "slice.py", line 17, in <module>
    test2 = f['data'][slice_obj]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 439, in __getitem__
    self.id.read(mspace, fspace, arr, mtype)
  File "h5d.pyx", line 179, in h5py.h5d.DatasetID.read (h5py/h5d.c:2479)
  File "_proxy.pyx", line 118, in h5py._proxy.dset_rw (h5py/_proxy.c:1300)
  File "_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread (h5py/_proxy.c:1051)
IOError: can't read data (Dataset: Read failed)

Does anyone know if this is just not possible with h5py? If it is not, is there an alternative way to slice in h5py, using objects or variables, instead of explicitly typing the slice like f['data'][0:3,2:5] as in my example?

Upvotes: 4

Views: 2614

Answers (1)

hpaulj
hpaulj

Reputation: 231355

Playing around with your example:

test2 = f['data']
print test2
print test2.shape
print test2[0:3,2:5]
print test2[slice(None,3,None),slice(2,5,None)]  # ok
print test2[slice_obj[0],slice_obj[1]]  # ok
print test2[tuple(slice_obj)]  # ok
print test2[[slice(None,3,None),slice(2,5,None)]]  # fail
print f['data'][tuple(slice_obj)] 3 ok

So it looks like h5py arrays can use slices, but cannot split a list into its elements. But it does take a tuple. My guess is that there is minor difference in how getitem is implemented.

You are doing advanced indexing. numpy doc says:

Advanced indexing is triggered when the selection object, obj,... a tuple with at least one sequence object.... when the selection object is not a tuple, it will be referred to as if it had been promoted to a 1-tuple, which will be called the selection tuple.

h5py may not be doing this promoting to tuple. Otherwise it appears to do advance indexing just fine.

Upvotes: 5

Related Questions