levanpon
levanpon

Reputation: 33

How to iterate tensorflow Dataset?

i'm trying to build a data pipelines from tfrecords, here is my code

def _parse_image_function(example_proto):
    keys_to_features = {
        'image/encoded': tf.io.FixedLenFeature((), tf.string),
        'image/format': tf.io.FixedLenFeature((), tf.string, default_value='jpeg'),
        'image/height': tf.io.FixedLenFeature([1], tf.int64),
        'image/width': tf.io.FixedLenFeature([1], tf.int64),
        'image/channels': tf.io.FixedLenFeature([1], tf.int64),
        'image/shape': tf.io.FixedLenFeature([3], tf.int64),
        'image/object/bbox/xmin': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/bbox/ymin': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/bbox/xmax': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/bbox/ymax': tf.io.VarLenFeature(dtype=tf.float32),
        'image/object/bbox/label': tf.io.VarLenFeature(dtype=tf.int64),
        'image/object/bbox/difficult': tf.io.VarLenFeature(dtype=tf.int64),
        'image/object/bbox/truncated': tf.io.VarLenFeature(dtype=tf.int64),
    }

    example = tf.io.parse_single_example(example_proto, keys_to_features)
    image = tf.io.decode_raw(example['image/encoded'], tf.int32)
    return image

Then, i get image after decode

    for img in train_ds:
        print(img.numpy())

But i got an error

tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __inference_Dataset_map__parse_image_function_78}} Input to DecodeRaw has length 286478 that is not a multiple of 4, the size of int32
     [[{{node DecodeRaw}}]] [Op:IteratorGetNextSync]

How could I fix it?

Upvotes: 0

Views: 681

Answers (1)

moodoki
moodoki

Reputation: 109

As mentioned in the comments, the error message is about decoding error, and not a problem with iteration. You are creating the dataset object and iterating through it correctly.

image = tf.io.decode_raw(example['image/encoded'], tf.int32) tells TensorFlow to decode the data stored in that key as a tensor of int32s. That is, raw binary values of 32-bit integers, for example, the contents of .data in a NumPy array with dtype=np.int32.

Since you have read in the binary contents of .jpg files, I'm assuming that you have binaries of JPEG images at that key value. For this case, you should be using the decode_jpeg method instead. You should be using:

image = tf.io.decode_jpeg(example['image/encoded']).

decode_jpeg also provides you with some options on how you wish the JPEG data to be decoded (e.g. grayscale only, chroma upsampling method). Full documentation of decode_jpeg is available here

Additionally, TensorFlow also provides a decode_image that automatically calls the correct supported image format decoder given the binary data. See docs here.

Upvotes: 1

Related Questions