tensorflow keras model fit training loss stuck on some runs

Question

I am building a neural network model using tensorflow/keras that has a simpleRNN layer followed by couple of dense layers. The model design is shown below

model = keras.Sequential()
model.add(layers.InputLayer(input_shape=(num_months, input_size, )))
model.add(layers.SimpleRNN(64))
model.add(layers.Dense(16, activation = 'relu'))
model.add(layers.Dense(1, activation = 'relu'))

model.compile(loss = 'mse', optimizer = tf.keras.optimizers.Adam(0.0001), metrics = ['mse'])

model.fit(features, target, epochs = 100, batch_size = 16)

I understand that there is randomness in weight initialization and the order of the samples. But I am noticing that if I run the model training process multiple times, there are instances where the training loss is just stuck and doesn't change at all for all the epochs. In this case the final trained model predicts zero for all the samples in the training data. I have attached a screen shot below.

There are other instances where the training loss improves with the each successive epochs. A screenshot for that is shown below

I am wondering why this may be happening and what would be a good strategy to address this problem. Is this just an issue with the dataset I have or could there be things done on the network design that could address the issue.

Answer 1

Please try again by removing the activation = 'relu' from the last Dense Layer of the model which is causing this issue.

I tried replicating the error and found the same issue.

model = tf.keras.Sequential()
model.add(layers.InputLayer(input_shape=(9,1)))
model.add(layers.SimpleRNN(64))
model.add(layers.Dense(16, activation = 'relu'))
model.add(layers.Dense(1,activation = 'relu'))

model.compile(loss = 'mse', optimizer = tf.keras.optimizers.Adam(0.0001), metrics='mse')

model.fit(train_features, train_labels, epochs = 50, batch_size = 16)

Output:

0/20 [==============================] - 0s 5ms/step - loss: 602.9218 - mse: 602.9218

    Epoch 48/50
    20/20 [==============================] - 0s 5ms/step - loss: 602.9217 - mse: 602.9217
    Epoch 49/50
    20/20 [==============================] - 0s 8ms/step - loss: 602.9217 - mse: 602.9217
    Epoch 50/50
    20/20 [==============================] - 0s 9ms/step - loss: 602.9216 - mse: 602.9216
    <keras.callbacks.History at 0x7fb74145b3d0>

Fixed code: (After removing the 'relu' activation layer)

    model = keras.Sequential()
    model.add(layers.InputLayer(input_shape=(9,1)))
    model.add(layers.SimpleRNN(64))
    model.add(layers.Dense(16, activation = 'relu'))
    model.add(layers.Dense(1))
    
    model.compile(loss = 'mse', optimizer = tf.keras.optimizers.Adam(0.0001), metrics='mse')
    
    model.fit(train_features, train_labels, epochs = 50, batch_size = 16)

Output:

Epoch 1/50
20/20 [==============================] - 2s 7ms/step - loss: 602.8611 - mse: 602.8611
Epoch 2/50
20/20 [==============================] - 0s 5ms/step - loss: 566.6978 - mse: 566.6978
Epoch 3/50
20/20 [==============================] - 0s 4ms/step - loss: 543.8058 - mse: 543.8058
Epoch 4/50
20/20 [==============================] - 0s 4ms/step - loss: 526.6135 - mse: 526.6135
Epoch 5/50
20/20 [==============================] - 0s 5ms/step - loss: 509.6300 - mse: 509.6300
Epoch 6/50
20/20 [==============================] - 0s 4ms/step - loss: 493.1002 - mse: 493.1002
Epoch 7/50
20/20 [==============================] - 0s 4ms/step - loss: 477.3309 - mse: 477.3309
Epoch 8/50
20/20 [==============================] - 0s 5ms/step - loss: 462.2016 - mse: 462.2016
Epoch 9/50
20/20 [==============================] - 0s 6ms/step - loss: 447.8385 - mse: 447.8385
Epoch 10/50
20/20 [==============================] - 0s 8ms/step - loss: 434.0708 - mse: 434.0708
Epoch 11/50
20/20 [==============================] - 0s 8ms/step - loss: 420.9923 - mse: 420.9923
Epoch 12/50
20/20 [==============================] - 0s 10ms/step - loss: 408.6603 - mse: 408.6603
Epoch 13/50
20/20 [==============================] - 0s 9ms/step - loss: 396.4588 - mse: 396.4588
Epoch 14/50
20/20 [==============================] - 0s 7ms/step - loss: 379.1953 - mse: 379.1953
Epoch 15/50
20/20 [==============================] - 0s 7ms/step - loss: 360.9505 - mse: 360.9505
Epoch 16/50
20/20 [==============================] - 0s 8ms/step - loss: 345.2476 - mse: 345.2476
Epoch 17/50
20/20 [==============================] - 0s 9ms/step - loss: 330.7779 - mse: 330.7779
Epoch 18/50
20/20 [==============================] - 0s 7ms/step - loss: 317.6154 - mse: 317.6154
Epoch 19/50
20/20 [==============================] - 0s 8ms/step - loss: 305.3287 - mse: 305.3287
Epoch 20/50
20/20 [==============================] - 0s 8ms/step - loss: 293.7218 - mse: 293.7218
Epoch 21/50
20/20 [==============================] - 0s 8ms/step - loss: 282.8405 - mse: 282.8405
Epoch 22/50
20/20 [==============================] - 0s 8ms/step - loss: 272.5468 - mse: 272.5468
Epoch 23/50
20/20 [==============================] - 0s 9ms/step - loss: 262.9255 - mse: 262.9255
Epoch 24/50
20/20 [==============================] - 0s 8ms/step - loss: 253.4659 - mse: 253.4659
Epoch 25/50
20/20 [==============================] - 0s 7ms/step - loss: 244.6598 - mse: 244.6598
Epoch 26/50
20/20 [==============================] - 0s 5ms/step - loss: 236.2224 - mse: 236.2224
Epoch 27/50
20/20 [==============================] - 0s 5ms/step - loss: 228.2639 - mse: 228.2639
Epoch 28/50
20/20 [==============================] - 0s 5ms/step - loss: 220.4474 - mse: 220.4474
Epoch 29/50
20/20 [==============================] - 0s 6ms/step - loss: 213.0748 - mse: 213.0748
Epoch 30/50
20/20 [==============================] - 0s 6ms/step - loss: 205.9706 - mse: 205.9706
Epoch 31/50
20/20 [==============================] - 0s 6ms/step - loss: 199.2270 - mse: 199.2270
Epoch 32/50
20/20 [==============================] - 0s 6ms/step - loss: 192.5672 - mse: 192.5672
Epoch 33/50
20/20 [==============================] - 0s 6ms/step - loss: 186.2284 - mse: 186.2284
Epoch 34/50
20/20 [==============================] - 0s 5ms/step - loss: 180.1651 - mse: 180.1651
Epoch 35/50
20/20 [==============================] - 0s 4ms/step - loss: 174.3560 - mse: 174.3560
Epoch 36/50
20/20 [==============================] - 0s 4ms/step - loss: 168.6014 - mse: 168.6014
Epoch 37/50
20/20 [==============================] - 0s 5ms/step - loss: 163.2568 - mse: 163.2568
Epoch 38/50
20/20 [==============================] - 0s 6ms/step - loss: 157.9677 - mse: 157.9677
Epoch 39/50
20/20 [==============================] - 0s 7ms/step - loss: 152.9507 - mse: 152.9507
Epoch 40/50
20/20 [==============================] - 0s 5ms/step - loss: 148.0877 - mse: 148.0877
Epoch 41/50
20/20 [==============================] - 0s 5ms/step - loss: 143.3548 - mse: 143.3548
Epoch 42/50
20/20 [==============================] - 0s 4ms/step - loss: 138.9760 - mse: 138.9760
Epoch 43/50
20/20 [==============================] - 0s 6ms/step - loss: 134.5997 - mse: 134.5997
Epoch 44/50
20/20 [==============================] - 0s 6ms/step - loss: 130.5038 - mse: 130.5038
Epoch 45/50
20/20 [==============================] - 0s 5ms/step - loss: 126.5234 - mse: 126.5234
Epoch 46/50
20/20 [==============================] - 0s 5ms/step - loss: 122.6731 - mse: 122.6731
Epoch 47/50
20/20 [==============================] - 0s 7ms/step - loss: 119.0524 - mse: 119.0524
Epoch 48/50
20/20 [==============================] - 0s 7ms/step - loss: 115.5618 - mse: 115.5618
Epoch 49/50
20/20 [==============================] - 0s 7ms/step - loss: 112.2575 - mse: 112.2575
Epoch 50/50
20/20 [==============================] - 0s 6ms/step - loss: 109.0311 - mse: 109.0311
<keras.callbacks.History at 0x7fb73cebe0b0>