can't get tf.train.shuffle_batch() work properly

Question

I've been working the whole day on this and I don't think another would make a difference !

I have an.png file, from which I've made >400 copies[ I got to use images with different shapes, but for now I just want to get this starting ]

here the code I use hopping to images as Tensors with the labels:

import tensorflow as tf
import os
import numpy
batch_Size =20
num_epochs = 100
files = os.listdir("Test_PNG")
files = ["Test_PNG/" + s for s in files]
files = [os.path.abspath(s) for s in files ]


def read_my_png_files( filename_queue):
    reader = tf.WholeFileReader()
    imgName,imgTensor = reader.read(filename_queue)
    img =  tf.image.decode_png(imgTensor,channels=0)
    # Processing should be add
    return img,imgName

def inputPipeline(filenames, batch_Size, num_epochs= None):
    filename_queue  = tf.train.string_input_producer(filenames, num_epochs=num_epochs,shuffle =True)
    img_file, label = read_my_png_files(filename_queue)
    min_after_dequeue = 100
    capacity = min_after_dequeue+3*batch_Size
    img_batch,label_batch = tf.train.shuffle_batch([img_file,label],batch_size=batch_Size,enqueue_many=True,
                                                    allow_smaller_final_batch=True, capacity=capacity,
                                                    min_after_dequeue =min_after_dequeue, shapes=[w,h,d])
    return img_batch,label_batch

images, Labels  = inputPipeline(files,batch_Size,num_epochs)

Based on what I've understand I should get 20 times images as tensors and their labels. when I run the code bellow here is what I've get:

    ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-08857195e465> in <module>()
     34     return img_batch,label_batch
     35 
---> 36 images, Labels  = inputPipeline(files,batch_Size,num_epochs)

<ipython-input-3-08857195e465> in inputPipeline(filenames, batch_Size, num_epochs)
     31     img_batch,label_batch = tf.train.shuffle_batch([img_file,label],batch_size=batch_Size,enqueue_many=True,
     32                                                     allow_smaller_final_batch=True, capacity=capacity,
---> 33                                                     min_after_dequeue =min_after_dequeue, shapes=[w,h,d])
     34     return img_batch,label_batch
     35 

c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\training\input.py in shuffle_batch(tensors, batch_size, capacity, min_after_dequeue, num_threads, seed, enqueue_many, shapes, allow_smaller_final_batch, shared_name, name)
   1212       allow_smaller_final_batch=allow_smaller_final_batch,
   1213       shared_name=shared_name,
-> 1214       name=name)
   1215 
   1216 

c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\training\input.py in _shuffle_batch(tensors, batch_size, capacity, min_after_dequeue, keep_input, num_threads, seed, enqueue_many, shapes, allow_smaller_final_batch, shared_name, name)
    767     queue = data_flow_ops.RandomShuffleQueue(
    768         capacity=capacity, min_after_dequeue=min_after_dequeue, seed=seed,
--> 769         dtypes=types, shapes=shapes, shared_name=shared_name)
    770     _enqueue(queue, tensor_list, num_threads, enqueue_many, keep_input)
    771     full = (math_ops.cast(math_ops.maximum(0, queue.size() - min_after_dequeue),

c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\ops\data_flow_ops.py in __init__(self, capacity, min_after_dequeue, dtypes, shapes, names, seed, shared_name, name)
    626         shared_name=shared_name, name=name)
    627 
--> 628     super(RandomShuffleQueue, self).__init__(dtypes, shapes, names, queue_ref)
    629 
    630 

c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\ops\data_flow_ops.py in __init__(self, dtypes, shapes, names, queue_ref)
    151     if shapes is not None:
    152       if len(shapes) != len(dtypes):
--> 153         raise ValueError("Queue shapes must have the same length as dtypes")
    154       self._shapes = [tensor_shape.TensorShape(s) for s in shapes]
    155     else:

ValueError: Queue shapes must have the same length as dtypes

I declared the shape bellow to use in tf.train.shuffle_batch function but I still have a shape error!

Any idea how may solve this?

Answer 1

Your issue was both from

• the enqueue_many=True argument,
• the shape of the shapes argument where the label dimension was not there.

So I would try with enqueue_many=False and shapes=[[h, w, c], []]).

Indeed, if you look at the shuffle_batch doc:

If enqueue_many is False, tensors is assumed to represent a single example. An input tensor with shape [x, y, z] will be output as a tensor with shape [batch_size, x, y, z].

If enqueue_many is True, tensors is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of tensors should have the same size in the first dimension. If an input tensor has shape [*, x, y, z], the output will have shape [batch_size, x, y, z].

But in your code, it seems you dequeue only one single file: img_file, label = read_my_png_files(filename_queue) and you pass it straight to the shuffle_batch function: img_batch,label_batch = tf.train.shuffle_batch([img_file,label], ...) so the * dimension is missing and TensorFlow is expecting that the first dimension of [img_file,label] is the number of examples.

Also bear in mind that the enqueue_many and the dequeue_many are independent; i.e.

• *: the number of examples you queue into the queue, is independent of
• batch_size: The new batch size pulled from the queue.