Summing multiple loss in keras

Question

I am training a model that has two outputs, I get losses for both outputs, out_1_loss and out_2_loss that do not add up to the loss that is also displayed. Why is this happening ?

my model:

my_mdl = Model(Input_,[out_1,out_2])

and uses two losses:

my_mdl.compile(optimizer = 'adam', 
               loss = {'out_1': 'binary_crossentropy',
                       'out_2': 'mse'})

When I run fit on the model:

history = my_mdl.fit(train_x, {'out_1': train_y, 'out_2': train_x},
                     epochs=100,
                     batch_size=256,
                     validation_data=(val_x, {'out_1': val_y, 'out_2': val_x})) 

This is the keras log from fit:

Answer:

I had a kernel_regularizer in some of my dense layers which were adding a loss term to the the total loss

Answer 1

According to information provided by you, I created your use case as follows (let me know if I have assumed something wrong):

print('Keras version: ', keras.__version__)
print('Backend TF version: ', tf.__version__)

# Sample model
Input_ = Input(shape=(10,))
h1 = Dense(5)(Input_)
out_1 = Dense(5,name='out_1')(h1)
out_2 = Dense(10,name='out_2')(h1)
my_mdl = Model(Input_,[out_1,out_2])
print(my_mdl.summary())

my_mdl.compile(optimizer = 'adam', 
               loss = {'out_1': 'binary_crossentropy',
                       'out_2': 'mse'})

#Data
train_x = np.random.rand(2560,10)
train_y = np.random.rand(2560,5)

val_x = np.random.rand(100,10)
val_y = np.random.rand(100,5)

#Training
history = my_mdl.fit(train_x, {'out_1': train_y, 'out_2': train_x},
                     epochs=5,
                     batch_size=256,
                     validation_data=(val_x, {'out_1': val_y, 'out_2': val_x})) 

Output

Keras version:  2.2.2
Backend TF version:  1.8.0
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 10)           0                                            
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 5)            55          input_1[0][0]                    
__________________________________________________________________________________________________
out_1 (Dense)                   (None, 5)            30          dense_1[0][0]                    
__________________________________________________________________________________________________
out_2 (Dense)                   (None, 10)           60          dense_1[0][0]                    
==================================================================================================
Total params: 145
Trainable params: 145
Non-trainable params: 0
__________________________________________________________________________________________________
None
Train on 2560 samples, validate on 100 samples
Epoch 1/5
2560/2560 [==============================] - 0s 81us/step - loss: 6.2931 - out_1_loss: 5.8374 - out_2_loss: 0.4558 - val_loss: 5.9223 - val_out_1_loss: 5.4799 - val_out_2_loss: 0.4424
Epoch 2/5
2560/2560 [==============================] - 0s 7us/step - loss: 6.1148 - out_1_loss: 5.7093 - out_2_loss: 0.4055 - val_loss: 5.7849 - val_out_1_loss: 5.3890 - val_out_2_loss: 0.3959
Epoch 3/5
2560/2560 [==============================] - 0s 7us/step - loss: 5.9467 - out_1_loss: 5.5803 - out_2_loss: 0.3663 - val_loss: 5.6516 - val_out_1_loss: 5.2935 - val_out_2_loss: 0.3581
Epoch 4/5
2560/2560 [==============================] - 0s 7us/step - loss: 5.8020 - out_1_loss: 5.4687 - out_2_loss: 0.3333 - val_loss: 5.5436 - val_out_1_loss: 5.2183 - val_out_2_loss: 0.3253
Epoch 5/5
2560/2560 [==============================] - 0s 13us/step - loss: 5.6643 - out_1_loss: 5.3595 - out_2_loss: 0.3048 - val_loss: 5.4206 - val_out_1_loss: 5.1241 - val_out_2_loss: 0.2964

And final loss is sum of out_1_loss and out_2_loss.