CODEWITHSUNDEEP
tensorflowdeep-learning
resh
resh

Reputation: 11

Tensorflow-loss not decreasing when training

I am using tensorflow object detection api for my own dataset I am facing some problem. I am using centos , with GPU Geforce 1080, 8 GB GPU memory, tensorflow 1.2.1 . I have 500 images in training set and 40 in test. I did the following steps and I have two problems. 1.I annotated my images using LabelImg tool 2.Created tfrecord successfully 3.I used ssd_inception_v2_coco.config. I modified the only path, no of class and I did not train from scratch, I used ssd_inception_v2_coco model checkpoints.

Problem 1: from step 0 until 3000, my loss has dramatically decreased but after that, it stays constant between 5 to 6 . Not getting how I reduce it but still my model able to detect required object. Here is my Tensorborad samples Even i tried for diffent model eg. faster_rcnn_inception_resnet_v2_atrous_coco after some steps loss stay constant between 1 and 2

Problem 2: according to a document I able to run eval.py but getting the following error: WARNING:root:The following classes have no ground truth examples: 0 after that program terminate. I try to run train.py and eval.py at the same time still same error. Please give me a suggestion. I am tensorflow beginner required suggestion.

Upvotes: 1

Views: 2052

Answers (1)

Derek Chow
Derek Chow

Reputation: 732

The loss curve you're seeing on Tensorboard is quite normal. Initially, the loss will drop very quickly, but will seemingly "bottom out" over time. Training is a slow process, you should see a steady drop over time after more iterations.

Upvotes: 2

Related Questions