CODEWITHSUNDEEP
pythonscikit-learngridsearchcv
Philipp Steiner
Philipp Steiner

Reputation: 133

make_scorer(roc_auc_score) not equal to predefined scorer 'roc_auc'

I've been using GridSearchCV to optimize some parameters for a binary classifier. I want to operate the classifier at a point where it barely makes any false positives but reaches still a high true positive rate. So in short: Optimize TPR while restricting FPR to 0 (or close to).

Therefore I wanted to a slightly adapted roc_auc_score as scorer argument in the GridSearchCV.

clf1= SVC() 

# define grid-space (obviously i would use a biger grid for the actual optimization)
grid1 = {'C':[1, 1000], 'kernel': ['poly'],  'degree' : [3], 'class_weight': ['balanced'], 'probability':[True]}

#define scoring function: Since we want to keep FPR = 0 we calculate the roc curve only between FPR = [0, 0.0001] (instead of [0, 1]
roc_spec = make_scorer(roc_auc_score, max_fpr=0.001)#define roc score for the unsave class

grid_clf_acc = GridSearchCV(clf1, param_grid = grid1 , scoring = roc_spec, n_jobs = -1, cv=cross_validation_folds)
grid_clf_acc.fit(X_train, y_train)

As you can see, I've adapted sklearn's standard roc_auc_score by setting it's max_fpr to 0.001.

If I now run the grid search, unfortunately the algorithm does not use multiple confidence thresholds anymore to compute the roc_score but uses only one confidence threshold instead.

On the other hand if I dont use the 'selfmade' scorer and use Gridsearch with the preimplemented roc_auc_score, the algorithm does indeed use multiple thresholds to compute the auc_roc_score.

grid_clf_acc = GridSearchCV(clf1, param_grid = grid1 , scoring = 'roc_auc', n_jobs = -1, cv=cross_validation_folds)

So somehow, the slightly adapted roc_auc_score has not the same capabilites as the original roc_auc_score. Is this a bug, or am I making a mistake when I define my own scorer?

(Remarks:

  1. In this example I've used max_fpr=0.001. Even if I set it to 1, it still does calculate the roc_auc score based on one threshold only.
  2. I also tried the two arguments of the make_scorer function (needs_thresh & or needs_proba) but neither of them solved the problem.

Finally I share an image, that shows two ROC's that I made to localize the problem. The left one shows an ROC for the classifier that was generated with multiple thresholds. The number on top is the calculated ROC score. This score did not match the score i've got in the GridSearch when using the customized scorer. However it did match the the score when I used the preimplemented scorer. On the right i plotted an ROC for the classifer that was generated with one threshold only (--> I've used predict instead of predict_prob). The calculated ROC did indeed match the calculated but "faulty" ROC_AUC score of the GridSearchCV when using the customized scorer. enter image description here

Upvotes: 0

Views: 651

Answers (1)

Philipp Steiner
Philipp Steiner

Reputation: 133

I have found my mistake. What finally worked was initializing the scorer as following:

roc_spec = make_scorer(roc_auc_score, max_fpr=0.001, needs_proba=True)

Then i also had to set probabilty=True in the SVC:

clf1= SVC(probability=True)

This made it work.

Upvotes: 1

Related Questions