A_toaster
Reputation: 1288
High val_acc, but low accuracy in practice
I have been following the code from this website to train a CNN that can distinguish between 10 monkey species. During the training phase I get decent results with 3 epochs:
Using TensorFlow backend.
Found 1097 images belonging to 10 classes.
Found 272 images belonging to 10 classes.
Epoch 1/3
46/46 [==============================] - 24s 521ms/step - loss: 1.4526 - acc: 0.5644 - val_loss: 0.5665 - val_acc: 0.9688
Epoch 2/3
46/46 [==============================] - 20s 445ms/step - loss: 0.5706 - acc: 0.8983 - val_loss: 0.4078 - val_acc: 0.9375
Epoch 3/3
46/46 [==============================] - 21s 449ms/step - loss: 0.3696 - acc: 0.9447 - val_loss: 0.4832 - val_acc: 0.8438
Then I wrote a prediction script that looks at a directory and makes a prediction from the saved model. I tried pointing it at both the training and validation folders, but both of the results seem correct about 16% of the time. Why does my val_acc seem high during training but when I actually use the model to make a prediction, it performs poorly?
Here is the prediction loop:
inputSize = (300, 400, 3)
for count, path in enumerate(allImages):
img= cv2.imread(path)
img = cv2.resize(img, (inputSize[0], inputSize[1]))
img = np.reshape(img, (1, inputSize[0], inputSize[1], inputSize[2]))
correct = path.split(os.path.sep)[-2] #get label from directory
classes = labels[model.predict(img)[0].argmax(axis=0)]
if args.show_predictions: #display prediction in a window
fontSize = 0.8
black = (0, 0, 0)
white = (255, 255, 255)
orig = cv2.resize(orig, (480, 480))
cv2.putText(orig, 'prediction: {}'.format(classes), (0, 20),
cv2.FONT_HERSHEY_SIMPLEX, fontSize, black, 10)
cv2.putText(orig, 'prediction: {}'.format(classes), (0, 20),
cv2.FONT_HERSHEY_SIMPLEX, fontSize, white, 2)
cv2.putText(orig, 'answer: {}'.format(correct), (0, orig.shape[0] - 10),
cv2.FONT_HERSHEY_SIMPLEX, fontSize, black, 10)
cv2.putText(orig, 'answer: {}'.format(correct), (0, orig.shape[0] - 10),
cv2.FONT_HERSHEY_SIMPLEX, fontSize, white, 2)
cv2.imshow('prediction', orig)
if cv2.waitKey(0) == ord('q'):
break
if args.verbose:
print('prediction: {} --> answer: {}'.format(classes, correct))
if classes == correct:
totalCorrect += 1
counter += 1
if count%verbosity == 0:
print('Predicting image {} out of {}'.format(count, len(allImages)))
print('total correct: {}/{} - {}%'.format(totalCorrect, counter, int(totalCorrect/counter * 100)))
cv2.destroyAllWindows()
Basically it reads the label from the directory and compares it to a label list I made earlier in the script.
Upon running the script with the training data (which I thought would be high since it was trained on it), I get:
Predicting image 600 out of 1097
Predicting image 650 out of 1097
Predicting image 700 out of 1097
Predicting image 750 out of 1097
Predicting image 800 out of 1097
Predicting image 850 out of 1097
Predicting image 900 out of 1097
Predicting image 950 out of 1097
Predicting image 1000 out of 1097
Predicting image 1050 out of 1097
total correct: 161/1097 - 14%
What makes the accuracy high during training, but low in practice?
EDIT: Running the script in verbose mode:
Predicting image 0 out of 272
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: bald_uakari --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: bald_uakari --> answer: bald_uakari
prediction: japanese_macaque --> answer: bald_uakari
prediction: japanese_macaque --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: white_headed_capuchin --> answer: bald_uakari
prediction: bald_uakari --> answer: bald_uakari
prediction: white_headed_capuchin --> answer: bald_uakari
prediction: japanese_macaque --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: japanese_macaque --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: bald_uakari
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
Predicting image 50 out of 272
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: common_squirrel_monkey --> answer: black_headed_night_monkey
prediction: nilgiri_langur --> answer: common_squirrel_monkey
prediction: japanese_macaque --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: white_headed_capuchin --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: pygmy_marmoset --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: nilgiri_langur --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: white_headed_capuchin --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: nilgiri_langur --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: common_squirrel_monkey
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: pygmy_marmoset --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: pygmy_marmoset --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
Predicting image 100 out of 272
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: japanese_macaque --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: japanese_macaque
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: japanese_macaque --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: mantled_howler
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: white_headed_capuchin --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: white_headed_capuchin --> answer: nilgiri_langur
Predicting image 150 out of 272
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: nilgiri_langur
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: japanese_macaque --> answer: patas_monkey
prediction: nilgiri_langur --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: nilgiri_langur --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: nilgiri_langur --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: japanese_macaque --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: pygmy_marmoset --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: patas_monkey
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
Predicting image 200 out of 272
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: pygmy_marmoset --> answer: pygmy_marmoset
prediction: japanese_macaque --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: japanese_macaque --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: nilgiri_langur --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: common_squirrel_monkey --> answer: pygmy_marmoset
prediction: white_headed_capuchin --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: nilgiri_langur --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: japanese_macaque --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: nilgiri_langur --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: patas_monkey --> answer: silvery_marmoset
prediction: japanese_macaque --> answer: silvery_marmoset
prediction: japanese_macaque --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: japanese_macaque --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: silvery_marmoset
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: white_headed_capuchin --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: white_headed_capuchin --> answer: white_headed_capuchin
Predicting image 250 out of 272
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: white_headed_capuchin --> answer: white_headed_capuchin
prediction: white_headed_capuchin --> answer: white_headed_capuchin
prediction: white_headed_capuchin --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: japanese_macaque --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: pygmy_marmoset --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: white_headed_capuchin --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
prediction: common_squirrel_monkey --> answer: white_headed_capuchin
total correct: 41/272 - 15%
Upvotes: 1
Views: 1422
Answers (1)
A_toaster
Reputation: 1288
So, as it turns out, I was using the wrong channel input mode. The pre-trained network I was using is called resnet50 from the keras library, and it uses the caffec style input.
The solution is to import preprocess_image() from keras, then run
img = preprocess_image(img)
Another factor was me loading images with cv2.imread. I'm uncertain but this may use the wrong channel format that resnet50 requires. The alternative is to use keras' load_img() function.
Conclusion: val_acc is generally correct during training. If your prediction accuracy is poor it is likely a data input problem!
Upvotes: 1
Related Questions
- Validation acc is very high in each fold but Test acc is very low
- Accuracy and val_accuracy don't change while training
- The val_accuracy is higher than training accuracy, and the test accuracy is very low compared to both val_accuracy and train_accuracy
- Good training accuracy and validaiton accuracy but poor prediction accuracy
- Validation accuracy very low but good training accuracy
- How to improve val_loss and val_acc
- Keras CNN val_accuracy, loss, accuracy stuck
- Keras Image Classification - Prediction accuracy on validation dataset does not match val_acc
- Cannot find the reason behind very low accuracy in my Convolutional Net model with Keras
- Keras: How come 'accuracy' is higher than 'val_acc'?