CODEWITHSUNDEEP
pythontensorflowmachine-learningkerasloss-function
Nitin
Nitin

Reputation: 7667

custom keras metric returns nan

I want to further bucket the output classes into fewer buckets in a classification problem. I have 4 output classes (i.e. 0, 1, 2, 3). But during training I also want to track the accuracy for 2 classes:

For that I created a new metric and compiled it with the model:

def new_classes_acc(y_true, y_pred):
  actual = tf.floor( y_true / 2 )
  predicted = tf.floor( y_pred / 2 )
  return K.categorical_crossentropy(actual, predicted)

Compiled it like so:

model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy', new_classes_acc])

I get nan as the accuracy value. What is the right way to do this?

Upvotes: 2

Views: 2035

Answers (1)

today
today

Reputation: 33410

Since there are 4 classes and you have set categorical_crossentropy as the loss, then the labels are one-hot encoded and would be in shape (n_samples, 4). Therefore, first you need to find the true and predicted classes using argmax function and then use floor function (further, you want to create a metric not a loss function; therefore you should not use K.categorical_crossentropy):

from keras import backend as K
import tensorflow as tf

def custom_metric(y_true, y_pred):
    tr = tf.floor(K.argmax(y_true, axis=-1) / 2)
    pr = tf.floor(K.argmax(y_pred, axis=-1) / 2)
    return K.cast(K.equal(tr, pr), K.floatx())

Now, let's test it. First we create a simple model and compile it:

model = Sequential()
model.add(Dense(4, activation='softmax', input_shape=(2,)))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy', custom_metric])

Then we create dummy data:

import numpy as np
data = np.array([1, 2]).reshape(1, 2)

and use our model to predict the labels for the given data:

print(model.predict(data))
# prints: [0.04662106, 0.8046941 , 0.07660434, 0.0720804 ]

So the second class has the highest probability and would be the predicted label. Now, considering the custom metric we defined, given either of [1, 0, 0, 0] or [0, 1, 0, 0] as true labels, the custom metric should gives us 1 (i.e. 100%). Let's confirm this:

true_labels = np.array([1, 0, 0, 0]).reshape(1,4)
print(model.evaluate(data, true_labels))    # gives: [3.0657029151916504, 0.0, 1.0]

The first element of the returned list corresponds to loss, the second one to accuracy and the third one to our custom metric. As you can see, the accuracy is zero (since the true class is class one but the predicted class is class two) and the custom metric is 1 as expected.

And the other case:

true_labels = np.array([0, 1, 0, 0]).reshape(1,4)
print(model.evaluate(data, true_labels))    # gives: [0.21729297935962677, 1.0, 1.0]

Here the accuracy is one (since both true and predicted classes are class two) and the custom metric is also one. You can further confirm this for the remaining two cases of [0, 0, 1, 0] and [0, 0, 0, 1] as true labels; both should return zero for the value of custom metric.

Bonus: what if the labels are sparse, i.e. 0, 1, 2 and 3? Then, you can use keras.np_utils.to_categorical() method to one-hot encode them and then use the custom metric defined above.

Upvotes: 4

Related Questions