keras: Assessing the ROC AUC of multiclass CNN

Question

I am using keras Sequential() API to build my CNN model for a 5-class problem. Since accurary is not a good metric for a multiclass problem, I have to assess other metrics measure to evaluate my model. Currently, I use sklearn's confusion_matrix and classification_report, but I would like to look into more metrics, so I decide to evaluate the ROC AUC, but I am not sure how this is done with keras, what modifications should I do to my code, etc.

Currently, this is how I build the model:

model = Sequential()
activ = 'relu'
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ, input_shape=(1, 100, 4)))
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ ))
model.add(MaxPooling2D(pool_size=(1, 2) ))

model.add(Conv2D(64, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(Conv2D(64, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(MaxPooling2D(pool_size=(1, 2)))

model.add(Flatten())
A = model.output_shape
model.add(Dense(int(A[1] * 1/4.), activation=activ)) 

model.add(Dense(5, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5, batch_size=64, shuffle=True, callbacks=callbacks,
                          validation_split=0.2)

Evaluation on the test:

Pred = model.predict(x_test, batch_size=32)
Pred_Label = np.argmax(Pred, axis=1)

test_acc = accuracy_score(y_test, Pred_Label)
ConfusionM = confusion_matrix(list(y_test), Pred_Label, labels=[0, 1, 2, 3, 4])
class_report = classification_report(list(y_test), Pred_Label, labels=[0, 1, 2, 3, 4])

To get the results, like below:

Confusion Matrix:   
[[ 2514  1040  2584  6690  1773]
 [  208   359    37   668   126]
 [ 1445  1156  1172  3438  1106]
 [ 3158  2014  2993 10185  1951]
 [  154    77    29   493   151]]

Classification Report: 
              precision    recall  f1-score   support

     Class:0       0.34      0.17      0.23     14601
     Class:1       0.08      0.26      0.12      1398
     Class:2       0.17      0.14      0.15      8317
     Class:3       0.47      0.50      0.49     20301
     Class:4       0.03      0.17      0.05       904

    accuracy                           0.32     45521
   macro avg       0.22      0.25      0.21     45521
weighted avg       0.35      0.32      0.32     45521

How do I add ROC AUC to my model metrics?

Answer 1

    TypeError                                 Traceback (most recent call last)
    ~\AppData\Local\Temp/ipykernel_6176/1137608719.py in <module>
         17     return roc_auc_score(y_test, y_pred, average=average)
         18 
    ---> 19 print('ROC AUC score:', multiclass_roc_auc_score(ytest, ypred))
         20 
         21 c_ax.legend()
    
    ~\AppData\Local\Temp/ipykernel_6176/1137608719.py in multiclass_roc_auc_score(y_test, y_pred, average)

     13     for (idx, c_label) in enumerate(target):
     14         fpr, tpr, thresholds = roc_curve(y_test[:,idx].astype(int), y_pred[:,idx])
---> 15         c_ax.plot(fpr, tpr, label = '%s (AUC:%0.2f)'  % (c_label, auc(fpr, tpr)))
     16     c_ax.plot(fpr, fpr, 'b-', label = 'Random Guessing')
     17     return roc_auc_score(y_test, y_pred, average=average)

TypeError: 'list' object is not callable

for this code:-

def multiclass_roc_auc_score(y_test, y_pred, average="macro"):
    lb = LabelBinarizer()
    lb.fit(y_test)
    y_test = lb.transform(y_test)
    y_pred = lb.transform(y_pred)

    for (idx, c_label) in enumerate(target):
        fpr, tpr, thresholds = roc_curve(y_test[:,idx].astype(int), y_pred[:,idx])
        c_ax.plot(fpr, tpr, label = '%s (AUC:%0.2f)'  % (c_label, auc(fpr, tpr)))
    c_ax.plot(fpr, fpr, 'b-', label = 'Random Guessing')
    return roc_auc_score(y_test, y_pred, average=average)

print('ROC AUC score:', multiclass_roc_auc_score(ytest, ypred))
    
c_ax.legend()
c_ax.set_xlabel('False Positive Rate')
c_ax.set_ylabel('True Positive Rate')
plt.show()

Answer 2

Another way to plot the ROC curve of the multiclass classifier is shown below. Let's walk with a toy problem, CIFAR10, a multiclass data set, consist of 10 different classes.

import tensorflow as tf
import numpy as np 

(x_train, y_train), (_, _) = tf.keras.datasets.cifar10.load_data()

# train set / data 
x_train = x_train.astype('float32') / 255
y_train = tf.keras.utils.to_categorical(y_train , num_classes=10)

print(x_train.shape, y_train.shape)
# (50000, 32, 32, 3) (50000, 10)

The model

input = tf.keras.Input(shape=(32,32,3))
efnet = tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3),
                               strides=(1, 1), input_shape=(32, 32, 3),
                               activation='relu')(input)

# Now that we apply global max pooling.
gap = tf.keras.layers.GlobalMaxPooling2D()(efnet)

# Finally, we add a classification layer.
output = tf.keras.layers.Dense(10, activation='softmax')(gap)

# bind all
func_model = tf.keras.Model(input, output)

Compile and Run

func_model.compile(
          loss      = tf.keras.losses.CategoricalCrossentropy(),
          metrics   = tf.keras.metrics.CategoricalAccuracy(),
          optimizer = tf.keras.optimizers.Adam())
# fit 
func_model.fit(x_train, y_train, batch_size=128, epochs=20, verbose = 1)

Get predicted label and ground truth label

ypred = func_model.predict(x_train)
ypred = ypred.argmax(axis=-1)
ypred
array([7, 9, 7, ..., 9, 1, 1])

ytrain = y_train.argmax(axis=-1)
ytrain
array([6, 9, 9, ..., 9, 1, 1])

Plot ROC Curve of the individual target.

import matplotlib.pyplot as plt 
from sklearn.preprocessing import LabelBinarizer
from sklearn.metrics import roc_curve, auc, roc_auc_score


target= ['airplane', 'automobile', 'bird', 'cat', 'deer',
          'dog', 'frog', 'horse', 'ship', 'truck']

# set plot figure size
fig, c_ax = plt.subplots(1,1, figsize = (12, 8))

# function for scoring roc auc score for multi-class
def multiclass_roc_auc_score(y_test, y_pred, average="macro"):
    lb = LabelBinarizer()
    lb.fit(y_test)
    y_test = lb.transform(y_test)
    y_pred = lb.transform(y_pred)

    for (idx, c_label) in enumerate(target):
        fpr, tpr, thresholds = roc_curve(y_test[:,idx].astype(int), y_pred[:,idx])
        c_ax.plot(fpr, tpr, label = '%s (AUC:%0.2f)'  % (c_label, auc(fpr, tpr)))
    c_ax.plot(fpr, fpr, 'b-', label = 'Random Guessing')
    return roc_auc_score(y_test, y_pred, average=average)


print('ROC AUC score:', multiclass_roc_auc_score(ytrain, ypred))

c_ax.legend()
c_ax.set_xlabel('False Positive Rate')
c_ax.set_ylabel('True Positive Rate')
plt.show()

ROC AUC score: 0.6868888888888889

Answer 3

You can add the below code before you compile the model. There are a variety of metrics, that you can explore with the keras library metrics.

METRICS = [keras.metrics.AUC(name='auc')]
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=METRICS)

Further, you can design a function in these similar lines that you have used the confusion matrix with the sklearn library function, and plot the results.

def plot_roc(Pred, Pred_Label):
  fp, tp, _ = sklearn.metrics.roc_curve(labels, predictions).

You should definitely check this out

Cheers.