CODEWITHSUNDEEP
pythonunicodeh5py
Doug
Doug

Reputation: 169

H5py Unicode support for dictionaries

So, with h5py there is no support for unicode in attributes. It throws the error: TypeError: No conversion path for dtype: dtype('

I have seen the recommended work-around which is to encode the strings before storing like so: (f.attrs['x'] = [a.encode('utf8') for a in mylist])

However, I do not understand how to encode them into strings when storing them in a dictionary.

My code is as follows:

with sess.as_default():

        if vae_checkpoint:
            print('Restoring VAE checkpoint: %s' % vae_checkpoint)
            saver.restore(sess, vae_checkpoint)

        nrof_images = len(image_list)
        nrof_batches = int(math.ceil(len(image_list) / args.batch_size))
        latent_vars = np.zeros((nrof_images, args.latent_var_size))
        attributes = np.zeros((nrof_images, nrof_attributes))
        for i in range(nrof_batches):
            start_time = time.time()
            latent_var_, attribs_, indices_ = sess.run([latent_var, attribs, indices])
            latent_vars[indices_,:] = latent_var_
            attributes[indices_,:] = attribs_
            duration = time.time() - start_time
            print('Batch %d/%d: %.3f seconds' % (i+1, nrof_batches, duration))
        # NOTE: This will print the 'Out of range' warning if the last batch is not full,
        #  as described by https://github.com/tensorflow/tensorflow/issues/8330

        # Calculate average change in the latent variable when each attribute changes
        attribute_vectors = np.zeros((nrof_attributes, args.latent_var_size), np.float32)
        for i in range(nrof_attributes):
            pos_idx = np.argwhere(attributes[:,i]==1)[:,0]
            neg_idx = np.argwhere(attributes[:,i]==-1)[:,0]
            pos_avg = np.mean(latent_vars[pos_idx,:], 0)
            neg_avg = np.mean(latent_vars[neg_idx,:], 0)
            attribute_vectors[i,:] = pos_avg - neg_avg

> filename = os.path.expanduser(args.output_filename)
>             print('Writing attribute vectors, latent variables and attributes to %s' % filename)
>             mdict = {'latent_vars':latent_vars, 'attributes':attributes, 
>                      'fields':fields, 'attribute_vectors':attribute_vectors }
>               with h5py.File(filename, 'w') as f:
>                 for key, value in iteritems(mdict):
>                     f.create_dataset(key, data=value)

How can you encode the strings when using the dictionary as above? Python 3.3

Thanks, Doug

Upvotes: 0

Views: 670

Answers (1)

nitely
nitely

Reputation: 2288

According to the docs, every string value must be encoded into byte-strings. For string values this means value.encode('utf-8') but for containers (mainly lists and tuples) of strings this means going through each value and encoding it.

In this case the value fields is a list of (unicode) strings. So, the solution is converting it into a list of byte-strings:

# ...
fields = [v.encode('utf-8') for v in fields]
mdict = {'latent_vars':latent_vars, 'attributes':attributes, 'fields':fields, 'attribute_vectors':attribute_vectors}
# ...

The rest of variables are numpy arrays of integers so they are ok.

Upvotes: 1

Related Questions