CODEWITHSUNDEEP
pythonmachine-learningkerasdeep-learning
amro_ghoneim
amro_ghoneim

Reputation: 535

how to correctly output precision, recall and f1score in keras?

I have a data set of images that I divided into Training and Testing folders, each divided into the two classes I am classifying. I use Keras generators to fit and evaluate the data. I found some resources online that I followed to implement precision, recall and f1-score metrics. Here is my Code:

class_mode = 'binary'
out_activation = 'sigmoid'
epochs = 1
mode = 'grayscale'
cat_or_bin = 'binary_crossentropy'
out_activation = 'sigmoid'
image_size = 224
batch = 128
channels = 1

def model_logistic():
    m = Sequential()
    m.add(Flatten(input_shape = (image_size, image_size, channels)))
    m.add(Dropout(0.2))
    m.add(Dense(out,activation=out_activation))
    return m


def recall_m(y_true, y_pred):
    y_true = K.ones_like(y_true)
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    all_positives = K.sum(K.round(K.clip(y_true, 0, 1)))

    recall = true_positives / (all_positives + K.epsilon())
    return recall


def precision_m(y_true, y_pred):
    y_true = K.ones_like(y_true)
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))

    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision


def f1_score(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2 * ((precision * recall) / (precision + recall + K.epsilon()))


train_generator = datagen.flow_from_directory(
    directory=dir,
    target_size=(image_size, image_size),
    color_mode=mode,
    batch_size=batch,
    classes = {'no_acc':0, 'acc':1},
    class_mode=class_mode,
    shuffle=True)

Test_generator = datagen.flow_from_directory(
    directory=dir_test,
    target_size=(image_size, image_size),
    color_mode=mode,
    batch_size=batch,
    classes = {'no_acc':0, 'acc':1},
    class_mode=class_mode,
    shuffle=True)

STEP_SIZE_TRAIN = train_generator.n // train_generator.batch_size
STEP_SIZE_TEST = test_generator.n // test_generator.batch_size

sgd = optimizers.sgd(learning_rate=0.0001, momentum=0.9, nesterov=True)

model.compile(loss=cat_or_bin, optimizer=sgd, metrics=['accuracy', f1_score, precision_m, recall_m])

    H = model.fit_generator(generator=train_generator,
                        steps_per_epoch=STEP_SIZE_TRAIN,
                        epochs=epochs,
                        verbose=1)

(loss,
accuracy,
f1_score, precision, recall) = model.evaluate_generator(test_generator, STEP_SIZE_TEST)

the output is the following for training and evaluation:

422/422 [==============================] - 384s 910ms/step - loss: 0.2392 - accuracy: 0.9303 - f1_score: 0.4661 - precision_m: 0.9502 - recall_m: 0.3174


2.7411930561065674
0.605730414390564
0.0
0.0
0.0

Why does it output zeroes for these metrics?

EDIT

Keras v2.3 actually now includes these metrics so I added them to my code as such:

from keras.metrics import Precision, Recall


model.compile(loss=cat_or_bin, optimizer=sgd, metrics=['accuracy', Precision(), Recall()])

However, the outputs are still zeroes for these metrics.

Upvotes: 0

Views: 682

Answers (1)

Rishit Dagli
Rishit Dagli

Reputation: 1006

I would advise you to do use callbacks, this would make it easier for you to keep track of these scores at the end of each epoch too-

Make a callback class-

class ModelMetrics(tf.keras.callbacks.Callback):

  def on_train_begin(self,logs={}):
    self.precisions=[]
    self.recalls=[]
    self.f1_scores=[]
  def on_epoch_end(self, batch, logs={}):

    y_val_pred=self.model.predict_classes(x_val)

    _precision,_recall,_f1,_sample=score(y_val,y_val_pred)  

    self.precisions.append(_precision)
    self.recalls.append(_recall)
    self.f1_scores.append(_f1)

Note for the above code to work you would also have to import score

And while fitting the network you could do something like this-

metrics = ModelMetrics()

history = model.fit(x_train, y_train,
              batch_size = batch_size,
              epochs = num_epochs,
              validation_data = (x_val, y_val),
              callbacks = [metrics])

print(metrics.precisions)

Upvotes: 1

Related Questions