Tensorflow Incompatible shapes error while training

Question

I have been training an object detector, was able to do this successfully with 1 class, now I have added 2 classes and a few more images to the dataset, I have regenerated xml to csv and tf records by deleting old ones, pbtxt file is correct and also deleted old checkpoints inside the training directory but when I run the training job like this:

python legacy/train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config

following this tutorial . I get the following error:

InvalidArgumentError (see above for traceback): Incompatible shapes: [2,1917] vs. [4,1]
 [[Node: Loss/Match/cond/mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match/cond/one_hot, Loss/Match/cond/Cast_2)]]

Answer 1

If you are training Mobilenet SSD following the Obstacle Detection API tutorial you could try commenting out the "ssd_random_crop" augmentation in your config file. This worked for me. Was initially pointed out by rky0930 here.