Is AUC a better metric than accuracy in case of imbalenced datasets in machine learning,If not which is the best metric?

Question

Is auc better in handling imbalenced data. As in most of the cases if I am dealing with imbalenced data accuracy is not giving correct idea. Even though accuracy is high, model has poor perfomance. If it's not auc which is the best measure to handle imbalenced data.

Answer 1

There is a difference between "point metrics" (precision, recall, F1 score) and metrics that apply to a whole classifier (like area under a ROC curve or area under a PR curve.)

Point metrics like precision, recall, and F1 score are calculated using a classifier's decisions. That means you need to have picked a hard decision threshold, e.g. anything above 0.5 is positive and anything below 0.5 is negative. (You could also have picked any other decision threshold between 0 and 1; choosing a different decision threshold will change the precision, recall, and F1 score that you calculate.)

You can also calculate metrics like area under the ROC curve ("AUC") and area under the precision-recall curve (AUPRC.) These metrics can be thought of as "averages" over different decision thresholds. You calculate these using the vector of predicted probabilities, NOT a vector of binary labels. Area under the ROC curve is super popular but is not very useful when your data is skewed to have a lot of true negatives. Area under the precision recall curve is a great metric to use when your data is skewed to have a lot of true negatives. For more information about how to calculate AUROC or AUPRC, and when to use each one, you can view this article on AUROC and this article on AUPRC.

Answer 2

Neither are good for imbalanced datasets. Use the area under the precision recall curve instead.

Answer 3

The great thing about imbalanced classes is not accuracy, because if one class has 1% of examples and the other has 99%, you can classify all examples as zero and still get 99% accuracy.

Considering the confusion matrix (below), you should also analyze Precision and Recall. These measures give you the total amount of false positives and false negatives.

Then you have to define which is your focus. Considering Predictive Maintenance, a false positive is a healthy machine classified as a failure, and a false negative is a machine with failure classified as healthy. You can have 99% accuracy and an excellent AUC and still get 0% precision.