CODEWITHSUNDEEP
keras
zhf061
zhf061

Reputation: 321

How to get loss for each sample within a batch in keras?

How can I get the loss for each sample within a batch? It seems that Keras does not provide any functions meeting the demand.

Upvotes: 10

Views: 7862

Answers (5)

chongkai Lu
chongkai Lu

Reputation: 502

Just set the batch size equal to 1. And then write a custom Callback for model.evaluate to collect each loss on_test_batch_end.

Upvotes: 0

anon767
anon767

Reputation: 51

I know this thread is a bit old, but I found a neat solution and maybe this helps someone:

import tensorflow as tf

# Suppose you have an already trained model
model = ...

loss = SparseCategoricalCrossentropy(reduction=tf.compat.v1.losses.Reduction.NONE) # The loss function is just an example, the reduction is the important one
model.compile(optimizer=model.optimizer, loss=loss) # keep your original optimizer

# And then you'll get each loss for each instance within a batch
model.evaluate(X,y, batch_size=128)

Upvotes: 5

user3731622
user3731622

Reputation: 5095

If you're not just interested in getting the loss for each sample during training, then you can use the evaluate method:

# assume images is already defined (e.g. as a numpy array)
#        images.shape = (n_images, n_rows, n_cols, n_channels
n_images = images.shape[0]
loss = n_images*[None]

for i in range(n_images):

    # note I'll use the same input for x & y parameters which is the case for an autoencoder
    loss[i] = model.evaluate(x=images[i:i+1,:,:,:],
                             y=images[i:i+1,:,:,:],
                             batch_size=None,
                             verbose=1,
                             steps=1
                             )

Upvotes: 0

Pedro Marques
Pedro Marques

Reputation: 2682

Keras always computes losses per sample. It has to do so in order to calculate the loss values that will be used as the basis of back propagation. The values are not typically exposed to the user other than as their average per batch but they are calculated by the loss function and then averaged for display purposes.

A very simple example model:

import tensorflow as tf
import tensorflow.keras.backend as K
keras = tf.keras

model = keras.models.Sequential([
    keras.layers.Input(shape=(4,)),
    keras.layers.Dense(1)
])

def examine_loss(y_true, y_pred):
  result = keras.losses.mean_squared_error(y_true, y_pred)
  result = K.print_tensor(result, message='losses')
  return result

model.compile('adam', examine_loss)
model.summary()

If you execute the follow test code:

import numpy as np
X = np.random.rand(100, 4)

def test_fn(x):
  return x[0] * 0.2 + x[1] * 5.0 + x[2] * 0.3 + x[3] + 0.6

y = np.apply_along_axis(test_fn, 1, X)

model.fit(X[0:4], y[0:4])

You should seem something like the following:

losses [23.2873611 26.1659927 34.1300354 6.16115761]

(Numbers will differ since they depend on random initialisation).

This may interest you or not depending on what you want to do with the individual losses. That was not clear at all from the initial question.

Upvotes: 7

kklocker
kklocker

Reputation: 166

I don't think keras provides this feature, because the loss for each individual sample inside a batch is never calculated. I think the loss is only calculated for an average over the batch.

One very useless (and time consuming) workaround could be to iterate over the training data and train on single samples.

Correct me if it turns out I'm wrong.

Upvotes: 0

Related Questions