How can you map values in a tf.data.Dataset using a dictionary

Question

Here is a simple use-case of a desired mapping. To map integer labels to one-hot encodings. I would like to mention that for this particular case one should use tf.one_hot. But I want to understand how you could map a dataset using a dictionary anyway.

import tensorflow as tf
import numpy as np

#CREATE A ONE-HOT ENCODING MAPPING
mike_labels = [164, 117, 132, 37, 66, 177, 225, 33, 28, 75, 7]
num_classes = len(mike_labels)
one_hots = np.eye(len(mike_labels))
one_hots = one_hots.tolist()
#used to convert labels to corresponding one-hot encoding
label_encoder = {orig: onehot for orig, onehot in zip(mike_labels, 
one_hots)}
print (label_encoder[164])
print (label_encoder[28])

#CREATE A FAKE DATASET
raw_data = [[164],[28],[132],[7]]
dataset = tf.data.Dataset.from_generator(lambda: raw_data, tf.float32, output_shapes=[None])

iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()

with tf.Session() as sess:
    print(sess.run(next_element))
    print(sess.run(next_element))

The code prints out 4 values. The first are the desired one-hot encodings directly taken from the dictionary. The second two printed values are the first 2 values in the dataset. Each element is shown to be a list containing a single float.

[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0]
[ 164.]
[ 28.]

The ideal answer will show how to change all values in the dataset to their corresponding one-hot encodings in the dictionary, USING the provided dictionary and WILL NOT use tf.one_hot.

Answer 1

The labels can be mapped using a lambda function. The dataset.map function calls the function for each element of the dataset. The lambda function in the mapping will call another function using tf.py_func.

tf.py_func allows the tensors to be treated as np arrays since the tensor cannot be fed to the dictionary. The return value of the function will be a list of floats, tf.py_func needs the datatype of each so this is given with a list comprehension:

dataset = dataset.map(lambda label: tf.py_func(practice_py_func, [label], [tf.float32 for i in range(num_classes)]))

The following function will be called. First we obtain a list from the received numpy array. This list contains a single element (the label). As such, we take the element at position 0 and use the dictionary to find the corresponding one-hot encoding. As tensorflow seems to throw a strange error that the received value is a double rather than the expected float, we then cast it to float32. The one-hot encoding is then returned.

def practice_py_func(arg1):
    temp = arg1.tolist() #convert the numpy array to a list
    l = label_encoder[temp[0]] #look up the encoding in the dictionary
    output = [np.float32(val) for val in l] #convert each value in the encoding to a float
    return output

The whole solution looks like this:

import tensorflow as tf
import numpy as np

#CREATE A ONE-HOT ENCODING MAPPING
mike_labels = [164, 117, 132, 37, 66, 177, 225, 33, 28, 75, 7]
num_classes = len(mike_labels)
one_hots = np.eye(len(mike_labels))
one_hots = one_hots.tolist()
#used to convert labels to corresponding one-hot encoding
label_encoder = {orig: onehot for orig, onehot in zip(mike_labels, one_hots)}
print (label_encoder[164])
print (label_encoder[28])

#CREATE A FAKE DATASET
raw_data = [[164],[28],[132],[7]]
dataset = tf.data.Dataset.from_generator(lambda: raw_data, tf.float32, output_shapes=[None])


def practice_py_func(arg1):
    temp = arg1.tolist() #convert the numpy array to a list
    l = label_encoder[temp[0]] #look up the encoding in the dictionary
    output = [np.float32(val) for val in l] #convert each value in the encoding to a float
    return output

dataset = dataset.map(lambda label: tf.py_func(practice_py_func, [label], [tf.float32 for i in range(num_classes)]))


iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()

with tf.Session() as sess:
    print(sess.run(next_element))
    print(sess.run(next_element))