CODEWITHSUNDEEP
validationkerasrecurrent-neural-networkmetricsloss-function
NOo07
NOo07

Reputation: 51

Keras validation loss and metric inconsistent

code:

import tensorflow as tf
import numpy as np
from keras.models import Sequential
from keras import layers
from keras.optimizers import Adam
import keras

I use random input data:

data_np = np.random.rand(6400,10)
target_np = np.random.rand(6400,)

Normalizing the data:

mean1 = data_np[:].mean(axis=0)
std1 = data_np[:].std(axis=0)  
    
data_np -= mean1
data_np /= std1
        
mean2 = target_np.mean(axis=0)
std2 = target_np.std(axis=0)
        
target_np -= mean2
target_np /= std2

Create RNN input with lookback:

lookback = 7

train_data = np.array([data_np[(i-lookback):i,:] for i in range(lookback,len(data_np))])
target_data = target_np[lookback:len(data_np)]

And then set up a simple RNN:

model = Sequential()
model.add(layers.SimpleRNN(64,
                     activation = 'relu',
                     return_sequences=False,
                     input_shape=(train_data.shape[1], train_data.shape[2])))
model.add(layers.Dense(1))

opt = Adam(learning_rate=0.1)

mae = tensorflow.keras.losses.MeanAbsoluteError()

model.compile(optimizer=opt, loss=mae, metrics=[mae])
history = model.fit(train_data, target_data,
                              steps_per_epoch=round(0.7*len(train_data))//64,
                              epochs=10,
                              shuffle=False,
                              validation_split=0.3,
                              validation_steps = round(0.3*len(train_data))//64,
                              verbose=1)

The output then looks like this:

Train on 3495 samples, validate on 1498 samples Epoch 1/10 54/54 [==============================] - 2s 38ms/step - loss: 0.7955 - mean_absolute_error: 0.7955 - val_loss: 0.0428 - val_mean_absolute_error: 22.6301 Epoch 2/10 54/54 [==============================] - 2s 30ms/step - loss: 0.7152 - mean_absolute_error: 0.7152 - val_loss: 0.0421 - val_mean_absolute_error: 22.2968

I would expect val_loss and val_mean_absolute_error to be the same. Moreover, the levels don't make much sense either. After 10 epochs, I get

Epoch 10/10 54/54 [==============================] - 2s 32ms/step - loss: 0.7747 - mean_absolute_error: 0.7747 - val_loss: 0.0409 - val_mean_absolute_error: 21.6337

If I calculate the mean absolute error manually:

N=len(data_np)

val_data = np.array([data_np[(i-lookback):i,:] for i in range(round(0.7*N),N)])
val_target = target_np[round(0.7*N):N]

model_output = model.predict(val_data)

model_output=[output[0] for output in model_output]
np.mean(abs(model_output-val_target))

0.940300949276649

This looks like a result that one could expect. However, it is not even close to either val_loss or val_mean_absolute_error. What is wrong here?

Upvotes: 3

Views: 562

Answers (1)

NOo07
NOo07

Reputation: 51

OK. I managed to solve the issue by consistently using tensorflow.keras. So, replacing

import tensorflow as tf
import numpy as np
from keras.models import Sequential
from keras import layers
from keras.optimizers import Adam
import keras

with

import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
import tensorflow.keras

(and corrected a couple of details in the original question)

Upvotes: 2

Related Questions