CODEWITHSUNDEEP
python-3.xvalidationtensorflowkerasneural-network
Albert
Albert

Reputation: 399

Manual & automatic validation losses are different

I use Tensorflow 1.14.0 and Keras 2.2.4. I am using the following code with a custom validation loss:

seedn=40
import numpy as np
np.random.seed(seedn)
import random
random.seed(seedn)
import tensorflow as tf
tf.set_random_seed(seedn)

from tensorflow.keras import optimizers
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Input, Dense, Activation
from tensorflow.keras.models import Model, Sequential

def EVM_cal(data,prediction):
    return np.sqrt(np.mean((prediction - data) ** 2) / np.mean(data ** 2))*100

def EVMper(data, prediction):
    return K.sqrt(K.mean((prediction-data) ** 2) / K.mean(data** 2)) * 100


x_train_s=np.reshape(np.arange(1,1201),(100,12))/18000;
x_test_s=np.reshape(np.arange(1201,1801),(50,12))/18000;

model = Sequential()
model.add(Dense(8, input_shape=(x_train_s.shape[1],)))
model.add(Activation('linear'))
model.add(Dense(x_train_s.shape[1],))
model.add(Activation('linear'))

model.compile(optimizer='adam', loss=EVMper)
fittingadam = model.fit(x_train_s, x_train_s, epochs=4,
                        validation_data=(x_test_s, x_test_s),verbose=2)

Val_Loss = np.array(fittingadam.history['val_loss'])
print('Last Val Loss (Automatic)= ', Val_Loss[-1])

myEVM = EVM_cal(x_test_s,  model.predict(x_test_s))
print('Last Val Loss (Manual) =', myEVM)

The output is:

Last Val Loss (Automatic)=  89.88273468017579
Last Val Loss (Manual) = 89.90268535029907

As we see, the manual and automatic validation losses are off about 0.02. Here, I have generated the test and train set as an example. But, for my real dataset, the manual and automatic validation losses are off about 2. Why they are different? How can I fix this?

Upvotes: 1

Views: 141

Answers (2)

Daniel Möller
Daniel Möller

Reputation: 86600

Not sure, but this might be a matter of batch size during evaluation. Different from usual losses, yours mixes values from many samples together. (The mean of two batches is not equal to result of the entire data)

You can try batch_size=50 (the size of the entire test data) to test.

You can also try to call the Keras metric yourself in a little model like this (may also be affected by batch size):

dataInput = Input((x_train_s.shape[1],))
predInput = Input((x_train_s.shape[1],))
output = Lambda(lambda x: EVMper(x[0], x[1]))([dataInput, predInput])

evaluator = Model([dataInput, predInput], output)
print(evaluator.predict([x_test_s, x_test_s]))

If the issue happens to be batch size, one solution, only for losses that are samplewise (don't mix samples as yours seems to mix), may be to leave the batch dimension for Keras to handle (must test). Don't take the mean of the batch dimension, only the other dimensions. The final shape of the loss should be (batch_size,).

Upvotes: 2

Ashmit Bhattarai
Ashmit Bhattarai

Reputation: 149

The reason, I suspect, is because the float32 is the default float-type in keras where are float64 is default float-type in numpy.

My suggestion would be to try cast_to_floatx

Something like:

from keras import backend as K

def EVM_cal(data,prediction):
    pred = K.cast_to_floatx(prediction) # assuming they are numpy arrays
    data = K.cast_to_floatx(data)
    return np.sqrt(np.mean((pred - data) ** 2) / np.mean(data ** 2))*100

See this for clarification on cast_to_floatx : https://keras.io/backend/#backend

Upvotes: 0

Related Questions