CODEWITHSUNDEEP
machine-learningkerasdeep-learningneural-networkconv-neural-network
MARCO LAGALLA
MARCO LAGALLA

Reputation: 257

Keras wrong accuracy using model.predict

I am using this code for a CNN

train_batches = ImageDataGenerator().flow_from_directory('dice_sklearn/train', target_size=(IMG_WIDTH, IMG_HEIGHT),
                                        classes=['1', '2', '3', '4', '5', '6'],
                                        batch_size=cv_opt['batch'],
                                        color_mode="grayscale")

test_batches = ImageDataGenerator().flow_from_directory('dice_sklearn/test', target_size=(IMG_WIDTH, IMG_HEIGHT),
                                       class_mode='categorical',
                                       batch_size=cv_opt['batch'],
                                       shuffle=False)
train_num = len(train_batches)
test_num = len(test_batches)

model = Sequential([
    Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=(IMG_WIDTH, IMG_HEIGHT, 1)),
    Conv2D(32, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Dropout(0.30),

    Conv2D(64, (3, 3), padding='same', activation='relu'),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Dropout(0.30),

    Conv2D(64, (3, 3), padding='same', activation='relu'),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Dropout(0.30),

    Flatten(),
    Dense(512, activation='relu'),
    Dropout(0.5),
    Dense(6, activation='softmax'),

])
print(model.summary())

model.compile(Adam(lr=cv_opt['lr']), loss='categorical_crossentropy', metrics=['accuracy'])
history = model.fit(train_batches, steps_per_epoch=train_num,
                              epochs=cv_opt['epoch'], verbose=2)

model.save('cnn-keras.h5')

test_batches.reset()
prediction = model.predict(test_batches, steps=test_num, verbose=1)
predicted_class = np.argmax(prediction, axis=1)
classes = test_batches.classes[test_batches.index_array]
accuracy = (predicted_class == classes).mean()
print("Final accuracy:", accuracy * 100)

Where

The output from the training phase (with model.fit) on the last line (last epoch) returns:

192/192 [==============================] - 98s 510ms/step - loss: 0.0514 - accuracy: 0.9818 - val_loss: 0.0369 - val_accuracy: 0.9833

But when I run this part of code:

test_batches.reset()
prediction = model.predict(test_batches, steps=test_num, verbose=1)
predicted_class = np.argmax(prediction, axis=1)
classes = test_batches.classes[test_batches.index_array]
accuracy = (predicted_class == classes).mean()
print("Final accuracy:", accuracy * 100)

I get an accuracy score very very low: (0.16). But if a plot the learning curves I can see that the test/validation curve (if in testing or in parameter tuning) both reach accuracies near 90%.

Am I using the model.predict in the wrong way?

Upvotes: 0

Views: 293

Answers (1)

Timbus Calin
Timbus Calin

Reputation: 15003

Your model is not overfitting. Steps 1 and 2 do not have to be implemented at all in order to solve your problem. In fact, it is even more wrong since the author states that in case of overfitting you need to add more layers, which is strongly advised against: when one has an overfitting model, the model needs to be made simpler, not more complex.

The solution to your issue lies in @Dr.Snoopy's answer : the order of the classes do not match.

My recommendation is to iterate manually through the entire test set, get the ground truth, get the prediction (ensure the same exact preprocessing on images like in the training set is applied on your test set images) before you feed them to your model.

Then, calculate your metrics. This will solve your problem.

For example, you could use the idea below:

    correctly_predicted = 0
    for image in os.scandir(path_to_my_test_directory):
        image_path = image.path
        image = cv2.imread(image_path)
        image = apply_the_same_preprocessing_like_in_training(image)
        #transform from (H,W,3) to (1,H,W,3) because TF + Keras predict only on batches
        image = np.expand_dims(image,axis=0)
        prediction_label = np.argmax(model.predict(image))
        if prediction_label == ground_truth_label:
              correctly_predicted+=1

Upvotes: 3

Related Questions