CODEWITHSUNDEEP
pythontensorflowkerasdeep-learning
whitepanda
whitepanda

Reputation: 488

How is the loss calculated in TensorFlow?

I was reading about the minimum squared error(MSE) in the TensorwFlow (tf) user document.

https://www.tensorflow.org/api_docs/python/tf/keras/metrics/mean_squared_error

When I hard-coded the MSE and print each loss calculated, I observe a different value than what is reported by tf.

import numpy as np
import random
import tensorflow as tf
from tensorflow import keras 

m = 200
n = 5
my_input=  np.random.random([m,n])
my_output =  np.random.random([m,1])


def obj(y_true,y_pred):
    loss = tf.keras.losses.mean_squared_error(y_true,y_pred)
    tf.print("\n")
    tf.print(tf.math.reduce_mean(loss),"\n")
    return (loss)



           

my_model = tf.keras.Sequential([  
    tf.keras.layers.Flatten(input_shape=(my_input.shape[1],)),
    tf.keras.layers.Dense(32, activation='softmax'),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1) 
])   
                    

my_model.compile(loss=obj ,optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001))
my_model.fit(my_input,  my_output, epochs=5, batch_size=20, verbose=1)

The loss values match in the first step, however, there is always a difference after that. Could someone explain me the mechanism behind the loss calculation?

Epoch 1/5


0.349255413 

 1/10 [==>...........................] - ETA: 3s - loss: 0.3493

0.449805915 



0.453376621 



0.500476539 



0.294586331 



0.269146353 



0.358534873 

 7/10 [====================>.........] - ETA: 0s - loss: 0.3822

Upvotes: 4

Views: 1958

Answers (2)

abdou_dev
abdou_dev

Reputation: 827

Mean squared error (MSE) is the most commonly used loss function for regression. The loss is the mean overseen data of the squared differences between true and predicted values, or writing it as a formula.

enter image description here

You can use MSE when doing regression, believing that your target, conditioned on the input, is normally distributed, and want large errors to be significantly (quadratically) more penalized than small ones.

According to your example , and as mentioned in the image above :

y_true is y and y_pred is the same as y~i , so it will calculate the loss every epoch in order to get the minimum value that means , y_true will be somehow closer to y_pred

Upvotes: 1

Marco Cerliani
Marco Cerliani

Reputation: 22031

By default, the training loss at the end of each epoch is the mean of the batch losses.

In your case, you reported the first seven steps of the first epoch with the corresponding batch losses. So the loss at the seventh step is computed as a simple mean of the losses from the previous steps:

(0.349255413 + 0.449805915 + 0.453376621 + 0.500476539 + 0.294586331 + 0.269146353 + 0.358534873) / 7 = 0.3822

Upvotes: 4

Related Questions