Confusion Matrix for Leave-One-Out Cross Validation in sklearn

Question

I know how to draw confusion matrix when I use the train and test split using sklearn but I do not know how to create the confusion matrix when I am using the leave-one-out cross validation as shown in this example:

# Evaluate using Leave One Out Cross Validation
import pandas
from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = pandas.read_csv(url, names=names)
array = dataframe.values
X = array[:,0:8]
Y = array[:,8]
num_folds = 10
num_instances = len(X)
loocv = model_selection.LeaveOneOut()
model = LogisticRegression()
results = model_selection.cross_val_score(model, X, Y, cv=loocv)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))

How should I create the confusion matrix for LOOCV in order to visualize the per-class accuracy?

Answer 1

Borrowing your method from here, you can work around the problem via creating a custom scorer that receives the metadata during the iterations. These metadata can be used to find: F1 Score, Precision, Recall, Accuracy as well as the Confusion Matrix!

---

Here we need another trick that is using GridSearchCV which accepts a custom scorer, so here we go!

---

Here is an example that you can work on more according to your absolute requirements:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer, accuracy_score, confusion_matrix
from sklearn.model_selection import GridSearchCV, StratifiedKFold


# Your method from the link you provided
def cm_analysis(y_true, y_pred, labels, ymap=None, figsize=(10,10)):
    if ymap is not None:
        y_pred = [ymap[yi] for yi in y_pred]
        y_true = [ymap[yi] for yi in y_true]
        labels = [ymap[yi] for yi in labels]
    cm = confusion_matrix(y_true, y_pred, labels=labels)
    cm_sum = np.sum(cm, axis=1, keepdims=True)
    cm_perc = cm / cm_sum.astype(float) * 100
    annot = np.empty_like(cm).astype(str)
    nrows, ncols = cm.shape
    for i in range(nrows):
        for j in range(ncols):
            c = cm[i, j]
            p = cm_perc[i, j]
            if i == j:
                s = cm_sum[i]
                annot[i, j] = '%.1f%%\n%d/%d' % (p, c, s)
            elif c == 0:
                annot[i, j] = ''
            else:
                annot[i, j] = '%.1f%%\n%d' % (p, c)
    cm = pd.DataFrame(cm, index=labels, columns=labels)
    cm.index.name = 'Actual'
    cm.columns.name = 'Predicted'
    fig, ax = plt.subplots(figsize=figsize)
    sns.heatmap(cm, annot=annot, fmt='', ax=ax)
    #plt.savefig(filename)
    plt.show()


# Custom Scorer
def my_scorer(y_true, y_pred):
    acc = accuracy_score(y_true, y_pred)
    # you can either save  y_true, y_pred and accuracy into a file
    # for later use with the info in clf.cv_results_
    # or plot the confusion matrix right here!
    # for labels, you can create a class attribute to make it more dynamic
    # i.e. changes automatically with every new dataset!
    cm_analysis(y_true, y_pred, labels=[0,1], ymap=None, figsize=(10, 10))
    # N.B as long as you have y_true and y_pred from every round here, you can
    # do with them all the metrics that want such as F1 Score, Precision, Recall, A
    # ccuracy and the Confusion Matrix!
    return acc


url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
df = pd.read_csv(url, names=names)
array = df.values
X = np.array(array[:,0:8])
Y = np.array(array[:,8]).astype(int)

# I'll make it two just for submitting the result here!
num_folds = 2
skf = StratifiedKFold(n_splits=num_folds, random_state=0)

# this is just a trick because the list contains 
# the default parameter only (i.e. useless)
param_grid = {'C': [1.0]}
model = LogisticRegression()
# create custom scorer
custom_scorer = make_scorer(my_scorer)
# pass it to the GridSearchCV
clf = GridSearchCV(model, param_grid, scoring=custom_scorer, cv=skf, return_train_score=True)
# Fit and Go
clf.fit(X,Y)

# cv_results_ is a dict with all CV results during the iterations!
# IDK, you may need it to combine its content with the metrics ..etc
print(clf.cv_results_)

Result

{'mean_score_time': array([0.09023476]), 'split0_train_score': 
 array([0.79166667]), 'mean_train_score': array([0.77864583]), 
'params': [{'C': 1.0}], 'std_test_score': array([0.01953125]), 
'mean_fit_time': array([0.00235796]), 
'param_C': masked_array(data=[1.0], mask=[False], fill_value='?',
dtype=object), 'rank_test_score': array([1], dtype=int32), 
'split1_test_score': array([0.7734375]), 
'std_fit_time': array([0.00032902]), 'mean_test_score': array([0.75390625]), 
'std_score_time': array([0.00237632]), 'split1_train_score': array([0.765625]), 
'split0_test_score': array([0.734375]), 'std_train_score': array([0.01302083])}

Split 0

Split 1

---

EDIT

If you strictly want LOOCV, then you can apply it in the above code, just replace StratifiedKFold by LeaveOneOut function; but bear in mind that LeaveOneOut will iterate around 684 times! so it's computationally very expensive. However, that would give you the confusion matrix in details during the iterations (i.e. metadata).

Nevertheless, if you are seeking the confusion matrix of the overall (i.e. final) process then you will still need to use the GridSearchCV but like follow:

......
loocv = LeaveOneOut()
clf = GridSearchCV(model, param_grid, scoring='accuracy', cv=loocv)
clf.fit(X,Y)

y_pred = clf.best_estimator_.predict(X)
cm_analysis(Y, y_pred, labels=[0, 1], ymap=None, figsize=(10,10))

Result