Tensorflow FIFO Queue is closed and has insufficient element

Question

I have problems concerning my input pipeline on Tensorflow> I am not sure I understand correctly the queues.

I first create two separates slice_input_producer that generates paths to images and labels

paths_queue_mines=tf.train.slice_input_producer([t_pathsMine,t_pathMask,t_labels],
                                                shuffle=True)
paths_queue_bckg=tf.train.slice_input_producer((t_pathBckg,),
                                               shuffle=True)

After a processing I combine everything in one image tensor and one label tensor that I add to a new queue

images_queue=tf.FIFOQueue(capacity=300,
                          dtypes=[tf.float32,tf.float32],
                          shapes=[(SHAPE,SHAPE,1),(SUBDIVISION,SUBDIVISION,5)],
                          name='images_queue')


enqueue_op=images_queue.enqueue((img_final,label_final))
N_threads=20
#create QueueRunner for filling images_queue
qr=tf.train.QueueRunner(images_queue,[enqueue_op]*N_threads)
tf.train.add_queue_runner(qr)

Then I can launch my training using images_queue.dequeueMany but after a few steps of training (between 4 and 160 with a batch size of 64) I get the following error :

    FIFOQueue '_2_Preprocessing/images_queue' is closed and has insufficient elements (requested 64, current size 44)
     [[Node: images_queue_DequeueMany = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](Preprocessing/images_queue, images_queue_DequeueMany/n)]]

Caused by op 'images_queue_DequeueMany', defined at:
  File "main.py", line 194, in <module>
    run_training()
  File "main.py", line 114, in run_training
    images,labels = images_queue.dequeue_many(FLAGS.batch_size)
  File "/home/thales/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/data_flow_ops.py", line 458, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/home/thales/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1328, in _queue_dequeue_many_v2
    timeout_ms=timeout_ms, name=name)
  File "/home/thales/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/home/thales/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/thales/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

OutOfRangeError (see above for traceback): FIFOQueue '_2_Preprocessing/images_queue' is closed and has insufficient elements (requested 64, current size 44)
     [[Node: images_queue_DequeueMany = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](Preprocessing/images_queue, images_queue_DequeueMany/n)]]

I don't understand why the queue is closed. Even if it has not enough images inside, the program should wait for it to be filled again but when I catch the exception and try again 3 seconds later, the same happens with the same number of images in the queue.

When looking into tensorflow doc for sliceInputProducer

num_epochs: (Optional.) An integer. If specified input_producer produces each row of input_tensor num_epochs times before generating an OutOfRange error. If not specified, input_producer can cycle through the rows of input_tensor an unlimited number of times.

So the slice input producers should not close after seeing all the data.

Is it my preprocessing that is too heavy so the queue cannot fill quickly enough and is closed after some time ?

Thank you for your help

Answer 1

Problem solved : it was an error that randomly appears during the preprocessing. When my images_queue tries to enqueue a new image and raise this exception, nothing is printed but the queue closes. Then later when I try to access the queue, it raises the OutOfRangeError