Keras autoencoder : validation loss > training loss - but performing well on testing dataset

Question

IN SHORT:

I have trained an Autoencoder whose validation loss is always higher than its training loss (see attached figure). I would think that this is a signal of overfitting. However, my Autoencoder performs well on the testing dataset. I was wondering if:

1) with reference to the architecture of the network, provided below, anyone could provide insights on how to reduce the validation loss (and how it is possible that the validation loss is much higher than the training one, despite the performance of the Autoencoder being good on the testing dataset);

2) if it is actually a problem that there is this gap between training and validation loss (when the performance on the testing dataset is actually good).

DETAILS:

I coded up my deep Autoencoder in Keras (code below). The architecture is 2001 (input layer) - 1000 - 500 - 200 - 50 - 200 - 500 - 1000 - 2001 (output layer). My samples are 1d functions of time. Each of them has 2001 time components. I have 2000 samples, which I split in 1500 for training, 500 for testing. Ot of the 1500 training samples, 20% of them (i.e. 300) are used as validation set. I normalize the training set removing the mean and dividing by the standard deviation. I use the mean and standard deviation of the training dataset to normalise the testing dataset as well.

I train the Autoencoder using Adamax optimizer and mean squared error as loss function.

from tensorflow.keras.layers import Input, Dense, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras import optimizers

import numpy as np
import copy


# data
data = # read my input samples. They are 1d functions of time and I have 2000 of them.
# Each function has 2001 time components

# shuffling data before training
import random
random.seed(4)
random.shuffle(data)

# split training (1500 samples) and testing (500 samples) dataset
X_train = data[:1500]
X_test = data[1500:]

# normalize training and testing set using mean and std deviation of training set
X_mean = X_train.mean()
X_train -= X_mean
X_std = X_train.std()
X_train /= X_std

X_test -= X_mean
X_test /= X_std


### MODEL ###

# Architecture

# input layer
input_shape = [X_train.shape[1]]
X_input = Input(input_shape)

# hidden layers

x = Dense(1000, activation='tanh', name='enc0')(X_input)
encoded = Dense(500, activation='tanh', name='enc1')(x)
encoded_2 = Dense(200, activation='tanh', name='enc2')(encoded)
encoded_3 = Dense(50, activation='tanh', name='enc3')(encoded_2)
decoded_2 = Dense(200, activation='tanh', name='dec2')(encoded_3)
decoded_1 = Dense(500, activation='tanh', name='dec1')(decoded_2)
x2 = Dense(1000, activation='tanh', name='dec0')(decoded_1)

# output layer
decoded = Dense(input_shape[0], name='out')(x2)

# the Model
model = Model(inputs=X_input, outputs=decoded, name='autoencoder')

# optimizer
opt = optimizers.Adamax()
model.compile(optimizer=opt, loss='mse', metrics=['acc'])
print(model.summary())

###################

### TRAINING ###

epochs = 1000
# train the model
history = model.fit(x = X_train, y = X_train,
                    epochs=epochs,
                    batch_size=100,
                    validation_split=0.2)  # using 20% of training samples for validation

# Testing 
prediction = model.predict(X_test)
for i in range(len(prediction)):
    prediction[i] = np.multiply(prediction[i], X_std) + X_mean

loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(epochs)
plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()
plt.close()

Answer 1

2) if it is actually a problem that there is this gap between training and validation loss (when the performance on the testing dataset is actually good).

This is just the generalization gap, i.e. the expected gap in the performance between the training and validation sets; quoting from a recent blog post by Google AI:

An important concept for understanding generalization is the generalization gap, i.e., the difference between a model’s performance on training data and its performance on unseen data drawn from the same distribution.

.

I would think that this is a signal of overfitting. However, my Autoencoder performs well on the testing dataset.

It is not, but the reason is not exactly what you think (let alone the fact that "well" is a highly subjective term).

The telltale signature of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.:

Your graph does not show such a behavior; also, notice the gap (pun intended) between the curves in the above plot (adapted from the Wikipedia entry on overfitting).

how it is possible that the validation loss is much higher than the training one, despite the performance of the Autoencoder being good on the testing dataset

There is absolutely no contradiction here; notice that your training loss is almost zero, which is not necessarily surprising in itself, but it would certainly be surprising if the validation loss were anywhere close to zero. And, again, "good" is a highly subjective term.

In other words, nothing in the info you have provided shows that there is something wrong with your model...