CODEWITHSUNDEEP
pythonkerasconv-neural-networkmultilabel-classification
glomba
glomba

Reputation: 173

Good accuracy but wrong prediction

I have trained a model with a very good val_accuracy, but the predictions are completely wrong. Unfortunately answers to similar questions didn't help me. My network has a mutli label problem. The end result is to predict the 3 best labels for each picture from an album.

This is my function, which makes the output with the predictions.

def recognition(path):
    class = np.array(classes)
    for picture in os.listdir(path):
        pic = os.path.join(path, picture)
        pic = image.load_img(pic, size)
        pic = image.img_to_array(pic)
        pic = pic/255

This is my CNN:

model = models.Sequential()
model.add(pretrained)
model.add(layers.Dropout(0.3))
model.add(layers.Flatten())
model.add(layers.Dropout(0.3))
model.add(layers.Dense(256, activation = 'relu'))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(7, activation = 'sigmoid'))

The val_acc is at 0.9527.

Upvotes: 0

Views: 1811

Answers (3)

Seyed_Ali_Mohtarami
Seyed_Ali_Mohtarami

Reputation: 1164

Note that the accuracy may be deceptive. One case is when the data is imbalanced. Assume there are a total of 600 samples, where 550 belong to the Positive class and just 50 to the Negative class. Since most of the samples belong to one class, the accuracy for that class will be higher than for the other. If the model made a total of 530/550 correct predictions for the Positive class, compared to just 5/50 for the Negative class, then the total accuracy is (530 + 5) / 600 = 0.8917. This means the model is 89.17% accurate. With that in mind, you might think that for any sample (regardless of its class) the model is likely to make a correct prediction 89.17% of the time. This is not valid, especially when you consider the Negative class for which the model performed badly.

The precision is calculated as the ratio between the number of Positive samples correctly classified to the total number of samples classified. magine a man who is trusted by others; when he predicts something, others believe him. The precision is like this man. When the precision is high, you can trust the model when it predicts a sample as Positive. Thus, the precision helps to know how the model is accurate when it says that a sample is Positive.

reference: Evaluating Deep Learning Models

Upvotes: 0

Sree
Sree

Reputation: 983

For multi label problems, I don't think accuracy is a good metric. Try precision and recall. Keras Functional model giving high validation accuracy but incorrect prediction This discussion might be useful for you.

Upvotes: 2

Nikhil Vasudeva
Nikhil Vasudeva

Reputation: 56

I don't think there's any issue with the sigmoid (provided that this is a multi-label task). I would advice you to look at your data, try to make sense of whats happening. Maybe your dataset is completely different from the real world data (look into covariate shift). Just make sure you are not leaking any information into the validation set. There's no easy way to debug what is happening if the issue is data related.

Upvotes: 0

Related Questions