CODEWITHSUNDEEP
pythonscikit-learnrandom-forestgrid-search
BabyBetulla
BabyBetulla

Reputation: 45

RandomizedSearchCV precision score doesn't match in Random Forest

I am trying to carry out some hyperparameters optimization on a random forest using RandomizedSearchCV. I set the scoring method as average precision. The rand_search.best_score_ is around 0.38 (a reasonable result for my dataset), but when I compute the same average precision score using rand_search.best_estimator_ the result is close to 1 (see below). 

clf = RandomForestClassifier()
randsearch = RandomizedSearchCV(clf,
                                scoring='average_precision',
                                param_distributions=parameters,
                                n_iter=1,
                                cv=StratifiedShuffleSplit(n_splits=10),
                                n_jobs=1, 
                                verbose=2)

randsearch.fit(X, y)
randomized_best_score = randsearch.best_score_

best_estimator_avg_precision = average_precision_score(y, 
                               randsearch.best_estimator_.predict(X))

best_estimator_avg_precision_probs = average_precision_score(y, 
                          randsearch.best_estimator_.predict_proba(X)[:, 1])

print(randomized_best_score)
print(best_estimator_avg_precision)
print(best_estimator_avg_precision_probs)

>>> randomized_best_score: 0.3836
>>> best_estimator_avg_precision: 0.983577210629
>>> best_estimator_avg_precision_probs: 1.0

Any idea why this can happen? What am i doing wrong?

Upvotes: 1

Views: 1345

Answers (1)

Vivek Kumar
Vivek Kumar

Reputation: 36599

There are multiple things to note here.

1) randsearch.best_score_ is the mean score for test folds over training data, so will not match when you use full data to calculate score.

2) The RandomizedSearchCV will be trained on (fitted) whole data after finding the best combination of params (The combination of params which produced the best_score_). So when you score on the same data, it is expected to give higher results.

3) When you specify scoring='average_precision', internally its converted to the following:

average_precision_scorer = make_scorer(average_precision_score,
                                       needs_threshold=True)

The param needs_threshold is True here, that means the scorer requires the output of predict_proba() or decision_function() from the estimator. So your second output best_estimator_avg_precision is not comparable.

And for your best_estimator_avg_precision_probs, do this to check if it makes any difference:-

best_estimator_avg_precision_probs = average_precision_score(y, 
                      randsearch.best_estimator_.predict_proba(X))

But still, the scores will be higher because of point 2.

Upvotes: 1

Related Questions