Why is training accuracy at 99% but the prediction accuracy at 81% on the same data?

Question

I looked up similar questions to the problem but I dont understand still why it gives a such a result. Is it normal for a model to train up to 99% accuracy, but when used to predict on the same exact data, it gives a lower accuracy, in this case, 81%? Shouldn't it return back 99% accuracy?

Furthermore, when I present new unseen data, the prediction accuracy is an abysmal 17%. Surely this cannot be right. I understand that the model when presented new data should be less than the model's accuracy, but no way as bad as 17%.

Here is the code for context. I put comments for easier reading:

# Step 1) Split Data into Training and Prediction Sets
num_split_df_at = int(0.75*len(df))
np_train_data = df.iloc[0:num_split_df_at, columns_index_list].to_numpy()
np_train_target = list(df.iloc[0:num_split_df_at, 4])
np_predict_data = df.iloc[num_split_df_at:len(df), columns_index_list].to_numpy()
np_predict_target = list(df.iloc[num_split_df_at:len(df), 4])

# Step 2) Split Training Data into Training and Validation Sets
x_train, x_test, y_train, y_test = train_test_split(np_train_data, np_train_target, random_state=0)


# Step 3) Reshape Training and Validation Sets to (49, 5)
# prints: "(3809, 245)"
print(x_train.shape)
# prints: "(1270, 245)"
print(x_test.shape)
x_train = x_train.reshape(x_train.shape[0], round(x_train.shape[1]/5), 5)
x_test = x_test.reshape(x_test.shape[0], round(x_test.shape[1]/5), 5)
y_train = np.array(y_train)- 1
y_test = np.array(y_test)- 1
# prints: "(3809, 49, 5)"
print(x_train.shape)
# prints: "[0 1 2 3 4 5 6 7 8 9]"
print(np.unique(y_train))
# prints: "10"
print(len(np.unique(y_train)))

input_shape = (x_train.shape[1], 5)

# Step 4) Run Model
adam = keras.optimizers.Adam(learning_rate=0.0001)
model = Sequential()
model.add(Conv1D(512, 5, activation='relu', input_shape=input_shape))
model.add(Conv1D(512, 5, activation='relu'))
model.add(MaxPooling1D(3))
model.add(Conv1D(512, 5, activation='relu'))
model.add(Conv1D(512, 5, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
    
model.fit(x_train, y_train, batch_size=128, epochs=150, validation_data=(x_test, y_test))
print(model.summary())
model.save('model_1')

# Step 5) Predict on Exact Same Trained Data - Should Return High Accuracy
np_train_data = np_train_data.reshape(np_train_data.shape[0], round(np_train_data.shape[1]/5), 5)
np_train_target = np.array(np_train_target)- 1
predict_results = model.predict_classes(np_train_data)
print(accuracy_score(predict_results, np_train_target))

# Step 6) Predict on Validation Set
np_predict_data = np_predict_data.reshape(np_predict_data.shape[0], round(np_predict_data.shape[1]/5), 5)
np_predict_target = np.array(np_predict_target)- 1
predict_results = model.predict_classes(np_predict_data)
print(accuracy_score(predict_results, np_predict_target))

Here are the prediction results:

My input data looks similar to this - 49 Days, 5 data points per each day:

My output possible classification results are:

[1 2 3 4 5 6 7 8 9 10] converted to [0 1 2 3 4 5 6 7 8 9] for "sparse_categorical_crossentropy"

Answer 1

this is because the training accuracy/loss of Keras models are calculated batch wise and then averaged (see here). instead the validation metrics/performance are computed simultaneously on all the data passed.

this is simply to verify in this dummy example. we train a NN and pass as valid data the same train data. in this way, we can compare (a) training acc, (b) validation acc, and (c) accuracy_score at the end of the train. as we can see (b) = (c) but (a) is different from (c) and (b) for the reason expressed above

timestamp, features, n_sample = 45, 2, 1000
n_class = 10
X = np.random.uniform(0,1, (n_sample, timestamp, features))
y = np.random.randint(0,n_class, n_sample)

model = Sequential()
model.add(Conv1D(8, 3, activation='relu', input_shape=(timestamp, features)))
model.add(MaxPooling1D(3))
model.add(Conv1D(8, 3, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(n_class, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
history = model.fit(X, y, batch_size=128, epochs=5, validation_data=(X, y))

history.history['accuracy'][-1] # (a)
history.history['val_accuracy'][-1] # (b)
accuracy_score(y, np.argmax(model.predict(X), axis=1)) # (c)