CODEWITHSUNDEEP
tensorflowkerasscikit-learndeep-learningconv-neural-network
Yogesh Riyat
Yogesh Riyat

Reputation: 131

Issues producing sklearn metrics for multi-label classification

I've implemented an EfficientNet pre-trained model on retinal (eye) images, and I can't figure out why my metrics aren't working! I'm open to using other metrics packages (keras?) if that is the issue.

# Loading a pretrained conv base model
input_shape = (256, 256, 3)
conv_base = EfficientNetB7(weights=None, include_top=False, input_shape=input_shape)

dropout_rate = 0.2
number_of_classes = 3

initial_learning_rate=2e-5
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate,
    decay_steps=100000,
    decay_rate=0.96,
    staircase=True
)

en_model = models.Sequential()
en_model.add(conv_base)
en_model.add(layers.GlobalMaxPooling2D(name='gap'))

# Avoid overfitting 
en_model.add(layers.Dropout(rate=dropout_rate, name='dropout_out'))

# Set number_of_classes to the number of your final predictions
en_model.add(layers.Dense(number_of_classes, activation='sigmoid', name='fc_out')) #replaced softmax with sigmoid
conv_base.trainable = False

en_model.compile(
    #loss='sparse_categorical_crossentropy',
    #loss='categorical_crossentropy',
    #optimizer=optimizers.RMSprop(learning_rate=2e-5),
    loss='binary_crossentropy',
    optimizer=optimizers.Adam(learning_rate=lr_schedule),
    metrics=['accuracy']
)

history = en_model.fit(
    train_generator,
    steps_per_epoch=10,
    epochs=100,
    validation_data=val_generator,
    #validation_steps=None,
    validation_freq=1,
    verbose=1,
    callbacks=[tensorboard_callbacks],
    use_multiprocessing=True,
    workers=4
)
print('Average test loss: ', np.average(history.history['loss']))

Metrics - this is retinal (eye) image data, and there are three classes/labels - diabetic retinopathy, glaucoma, and other. I'll show the code for the first class, where you can see that the numbers are the same for each metric, and the confusion matrix has zeros. I can't figure out what is going on!

from sklearn.metrics import confusion_matrix, accuracy_score, f1_score, precision_score, recall_score

# y_true are the labels from the validation generator; we have three labels (DR, glaucoma, other)
print(f'Accuracy = {accuracy_score(val_generator.labels[:,0],[round(x) for x in val_pred[:,0]])}')
print(f"F1 = {f1_score(val_generator.labels[:,0],[round(x) for x in val_pred[:,0]], average='micro')}")
print(f"Precision = {precision_score(val_generator.labels[:,0],[round(x) for x in val_pred[:,0]], average='micro')}")
print(f"Recall = {recall_score(val_generator.labels[:,0],[round(x) for x in val_pred[:,0]], average='micro')}")
print('Confusion matrix =')
confusion_matrix(val_generator.labels[:,0],[round(x) for x in val_pred[:,0]])

Output

Accuracy = 0.7807953443258971
F1 = 0.7807953443258971
Precision = 0.7807953443258971
Recall = 0.7807953443258971
Confusion matrix =
array([[805,   0],
       [226,   0]])

The same code above (replacing 0's with 1's and 2's) produces these results:

Accuracy = 0.8244422890397672
F1 = 0.8244422890397672
Precision = 0.8244422890397672
Recall = 0.8244422890397672
Confusion matrix =
array([[850,   0],
       [181,   0]])

Accuracy = 0.6876818622696411
F1 = 0.6876818622696411
Precision = 0.6876818622696411
Recall = 0.6876818622696411
Confusion matrix =
array([[  0, 322],
       [  0, 709]])

Upvotes: 0

Views: 315

Answers (1)

AloneTogether
AloneTogether

Reputation: 26698

Keras and Tensorflow have already a lot of the metrics, if not all, that you mentioned if you are looking for alternatives:


!pip install -U tensorflow-addons
import tensorflow as tf

from tensorflow.python.keras.metrics import Accuracy
from tensorflow.python.keras.metrics import Recall
from tensorflow.python.keras.metrics import Precision
from tensorflow.python.keras.metrics import AUC
import tensorflow_addons as tfa

tf.math.confusion_matrix
tfa.metrics.F1Score

Regarding your Sigmoid activation function, you mentioned you have 3 classes; why not consider using a Softmax activation function which is actually used for multiclass logistic regression? You would also have to replace your binary_crossentropy loss function with categorical_crossentropy or sparse_categorical_crossentropy depending on how your labels are encoded. If you have a multi-label classification problem aiming to predict zero or more class labels (0, 1), then binary_crossentropy is the right choice for you.

Upvotes: 1

Related Questions