Reputation: 35
I am comparing two unsupervised models that predict a binary class, each with a different threshold parameter (p for model A and g for model B, ranging from 0 to 1). You can see that Model B has no monotonic ordering in PR trade-offs (correct behavior due to some approximations that I am researching) but generally fits closer to an ideal model at TPR=1 and FPR=0 (yellow dot).
My "issue" is that model B (regardless of value-setting g) never reaches a theoretical FPR=1. This is not bad, as no matter the value of g, there is always some TN in the denominator. However, if I compute the ROC-AUC (using numpy.trapz in Python) of the overall discriminative power, Model B is losing.
I thought about smoothing the curve, but I am not sure it is the right thing.
How can I "academically correct" compare both models using AUC ROC? I want to avoid cherry-picking the new proposed Model B :-)
Upvotes: 0
Views: 47