Reputation: 321
How can I get the loss for each sample within a batch? It seems that Keras does not provide any functions meeting the demand.
Upvotes: 10
Views: 7862
Reputation: 502
Just set the batch size equal to 1. And then write a custom Callback for model.evaluate to collect each loss on_test_batch_end.
Upvotes: 0
Reputation: 51
I know this thread is a bit old, but I found a neat solution and maybe this helps someone:
import tensorflow as tf
# Suppose you have an already trained model
model = ...
loss = SparseCategoricalCrossentropy(reduction=tf.compat.v1.losses.Reduction.NONE) # The loss function is just an example, the reduction is the important one
model.compile(optimizer=model.optimizer, loss=loss) # keep your original optimizer
# And then you'll get each loss for each instance within a batch
model.evaluate(X,y, batch_size=128)
Upvotes: 5
Reputation: 5095
If you're not just interested in getting the loss for each sample during training, then you can use the evaluate
method:
# assume images is already defined (e.g. as a numpy array)
# images.shape = (n_images, n_rows, n_cols, n_channels
n_images = images.shape[0]
loss = n_images*[None]
for i in range(n_images):
# note I'll use the same input for x & y parameters which is the case for an autoencoder
loss[i] = model.evaluate(x=images[i:i+1,:,:,:],
y=images[i:i+1,:,:,:],
batch_size=None,
verbose=1,
steps=1
)
Upvotes: 0
Reputation: 2682
Keras always computes losses per sample. It has to do so in order to calculate the loss values that will be used as the basis of back propagation. The values are not typically exposed to the user other than as their average per batch but they are calculated by the loss function and then averaged for display purposes.
A very simple example model:
import tensorflow as tf
import tensorflow.keras.backend as K
keras = tf.keras
model = keras.models.Sequential([
keras.layers.Input(shape=(4,)),
keras.layers.Dense(1)
])
def examine_loss(y_true, y_pred):
result = keras.losses.mean_squared_error(y_true, y_pred)
result = K.print_tensor(result, message='losses')
return result
model.compile('adam', examine_loss)
model.summary()
If you execute the follow test code:
import numpy as np
X = np.random.rand(100, 4)
def test_fn(x):
return x[0] * 0.2 + x[1] * 5.0 + x[2] * 0.3 + x[3] + 0.6
y = np.apply_along_axis(test_fn, 1, X)
model.fit(X[0:4], y[0:4])
You should seem something like the following:
losses [23.2873611 26.1659927 34.1300354 6.16115761]
(Numbers will differ since they depend on random initialisation).
This may interest you or not depending on what you want to do with the individual losses. That was not clear at all from the initial question.
Upvotes: 7
Reputation: 166
I don't think keras provides this feature, because the loss for each individual sample inside a batch is never calculated. I think the loss is only calculated for an average over the batch.
One very useless (and time consuming) workaround could be to iterate over the training data and train on single samples.
Correct me if it turns out I'm wrong.
Upvotes: 0