CODEWITHSUNDEEP
pythonscikit-learn
monkeybiz7
monkeybiz7

Reputation: 5128

Why are the grid_scores_ higher than the score for full training set? (sklearn, Python, GridSearchCV)

I'm building a logistic regression model as follows: 

cross_validation_object = cross_validation.StratifiedKFold(Y, n_folds = 10)
scaler = MinMaxScaler(feature_range = [0,1])
logistic_fit = LogisticRegression()

pipeline_object = Pipeline([('scaler', scaler),('model', logistic_fit)])

tuned_parameters = [{'model__C': [0.01,0.1,1,10],
                    'model__penalty': ['l1','l2']}]

grid_search_object = GridSearchCV(pipeline_object, tuned_parameters, cv = cross_validation_object, scoring = 'roc_auc')

I looked at the roc_auc score for the best estimator: 

grid_search_object.best_score_
Out[195]: 0.94505225726738229

However, when I used the best estimator to score the full training set, I got a worse score:

grid_search_object.best_estimator_.score(X,Y)
Out[196]: 0.89636762322433028

How can this be? What am I doing wrong?

Edit: Nevermind. I'm an idiot. grid_search_object.best_estimator_.score calculates accuracy, not auc_roc. Right?

But if that is the case, how does GridSearchCV compute the grid_scores_? Does it pick the best decision threshold for each parameter, or is the decision threshold always at 0.5? For area under the ROC curve, decision threshold doesn't matter, but it does for say, f1_score.

Upvotes: 0

Views: 1570

Answers (1)

eickenberg
eickenberg

Reputation: 14377

If you evaluated the best_estimator_ on the full training set it is not surprising that the scores are different from the best_score_, even if the scoring methods are the same:

The best_score_ is the average over your cross-validation fold scores of the best model (best in exactly that sense: scores highest on average over folds).

When scoring on the whole training set, your score may be higher or lower than this. Especially if you have some sort of temporal structure in your data and you are using the wrong data splitting, scores on the full set can be worse.

Upvotes: 2

Related Questions