Model-evaluation error using cross-validation - average_precision_score

Question

So I have ran the following random forest grid search using balanced_accuracy as my scoring:

# define the parameter grid
param_grid = [
        {'criterion': ['gini', 'entropy'],   # try different purity metrics in building the trees
         'max_depth': [2, 5, 8, 10, 15, 20],    # vary the max_depth of the trees in the ensemble
        'n_estimators': [10, 50, 100, 200],   # vary the number of trees in the ensemble
        'max_samples': [0.4, 0.7, 0.9]}     # vary how many samples each tree is built with
]

# setup the Random Forest model with all arguments as default
model = RandomForestClassifier()

# pass the model and the param_grid to the grid search, and use 5 folds with 'accuracy' as the scoring measure
grid_search = GridSearchCV(model, param_grid, cv = 5, scoring = 'balanced_accuracy')

# fit the grid search to the training set
grid_search.fit(X_smote, y_smote)

# return best model
rf_best = grid_search.best_estimator_

# return the hyperparameter values of the best model
print(grid_search.best_params_)

# use the best model to make predictions on the test set
y_pred = rf_best.predict(X_test)

# compute the test set accuracy of the best model
print("accuracy: ", accuracy_score(y_test,y_pred))
print("f1: ", f1_score(y_test, y_pred, pos_label='Listed'))
print("precision: ", precision_score(y_test, y_pred, pos_label='Listed'))
print("recall: ", recall_score(y_test, y_pred, pos_label='Listed'))

Which produces the following scores:

{'criterion': 'gini', 'max_depth': 20, 'max_samples': 0.7, 'n_estimators': 100}
accuracy:  0.6547231270358306
f1:  0.7612612612612613
precision:  0.9260273972602739
recall:  0.6462715105162524

I want to use the average_precision scoring parameter as this better suits my use case, so I update my syntax to the following:

from sklearn.metrics import average_precision_score
# define the parameter grid
param_grid = [
        {'criterion': ['gini', 'entropy'],   # try different purity metrics in building the trees
         'max_depth': [2, 5, 8, 10, 15, 20],    # vary the max_depth of the trees in the ensemble
        'n_estimators': [10, 50, 100, 200],   # vary the number of trees in the ensemble
        'max_samples': [0.4, 0.7, 0.9]}     # vary how many samples each tree is built with
]

# setup the Random Forest model with all arguments as default
model = RandomForestClassifier()

# pass the model and the param_grid to the grid search, and use 5 folds with 'accuracy' as the scoring measure
grid_search = GridSearchCV(model, param_grid, cv = 5, scoring = 'average_precision')

# fit the grid search to the training set
grid_search.fit(X_smote, y_smote)

# return best model
rf_best = grid_search.best_estimator_

# return the hyperparameter values of the best model
print(grid_search.best_params_)

# use the best model to make predictions on the test set
y_pred = rf_best.predict(X_test)

# compute the test set accuracy of the best model
print("accuracy: ", accuracy_score(y_test,y_pred))
print("f1: ", f1_score(y_test, y_pred, pos_label='Listed'))
print("precision: ", precision_score(y_test, y_pred, pos_label='Listed'))
print("recall: ", recall_score(y_test, y_pred, pos_label='Listed'))

However I receive the following error:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\metrics\_ranking.py in average_precision_score(y_true, y_score, average, pos_label, sample_weight)
    211         if len(present_labels) == 2 and pos_label not in present_labels:
    212             raise ValueError("pos_label=%r is invalid. Set it to a label in "
--> 213                              "y_true." % pos_label)
    214     average_precision = partial(_binary_uninterpolated_average_precision,
    215                                 pos_label=pos_label)

ValueError: pos_label=1 is invalid. Set it to a label in y_true.

Why cant I use average_precision on my code like I have done with balanced_accuracy. Is there something I should be doing different?

Answer 1

No idea what your dataset is like or where exactly is the error in your code. Too many redundant parts.

If the purpose is to use average precision score as stated, then you can use make_scorer, assuming your labels are binary, 0/1 like in example below:

from sklearn.datasets import make_blobs
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

param_grid = [
        {'criterion': ['gini', 'entropy'],   
         'max_depth': [2,5],    
        'n_estimators': [200],   
        'max_samples': [0.8]}]


X, y = make_blobs(n_samples=[80,20], centers=None, n_features=5,
cluster_std = 3.5,random_state=0)     

model = RandomForestClassifier(random_state=42)
grid_search_acc = GridSearchCV(model, param_grid, cv = 5, scoring = 'balanced_accuracy')

grid_search_acc.fit(X, y)

grid_search_acc.best_score_
0.75625

Balanced accuracy works, to make it work for average precision:

from sklearn.metrics import average_precision_score, make_scorer
ap_score = make_scorer(precision_score, greater_is_better=True, pos_label=1)

grid_search_prec = GridSearchCV(model, param_grid, cv = 5, scoring = ap_score)
grid_search_prec.fit(X, y)

grid_search_prec.best_score_
0.9333333333333332