CODEWITHSUNDEEP
rmachine-learningclassificationdata-mining
MK Baowaly
MK Baowaly

Reputation: 11

Evaluation metric in machine learning

For a binary classification problem as an example below, how logical will it be if I use "Accuracy" as metric in training and find AUC score using ROCR package? Or should I use "ROC" as metric always for computing AUC score? Dataset is imbalanced.

control <- trainControl(method="cv", number=5)

fit <- train(diabetes~., data=PimaIndiansDiabetes, method="gbm", metric="Accuracy", trControl=control)

Upvotes: 1

Views: 750

Answers (2)

Aditya
Aditya

Reputation: 1058

I think if you want to use accuracy then you should first make your dataset balanced using techniques like upsampling or downsampling. For interpretability, AUC is better compared to accuracy.

Upvotes: 0

Charan P
Charan P

Reputation: 135

For evaluating any classifier performance the most basic metric/bench-mark is the confusion matrix as Accuracy, precision, recall, F-measure, ROC and AUC all stem from the confusion matrix. Coming to your question, is accuracy as a performance metric enough to judge a binary classifier? The answer is "Yes", only if the data distribution is balanced, i.e there are equal number of class-1 and class-2 objects. But if data is imbalanced (as in your case) then it is a big NO to use accuracy as the key performance metric. A simple example to debunk the attractiveness/fallacy of accuracy is: Consider *** phone company wants to check the number of defective cell phones it produces. Now on an average there are 10 bad phones in every 1000 and the classifier built never catches a defective/bad phone in the 1000 phones it has sampled, then the accuracy of the classifier is still 99% as TP = 990 and TN = 0 and accuracy = TP+TN/1000. So the best or atleast the bench-mark to evaluate performance is F-score of each class and from there on it could be taken to the next level of plotting the ROC curves and evaluating the AUC.

Upvotes: 1

Related Questions