M B
M B

Reputation: 21

Training and evaluating accuracies different in keras LSTM model: why does evaluation not produce same results as during training?

We are building an LSTM for modeling a physical optical procees.

So far, I have produced the following code in python using Keras with Tensorflow backend.

#Define model
model = Sequential()
model.add(LSTM(128, batch_size=BATCH_SIZE, input_shape=(train_x.shape[1],train_x.shape[2]), return_sequences=True, stateful=False ))#,,return_sequences=Tru# stateful=True 
model.add(Dense(2, activation='softmax'))
opt = tf.keras.optimizers.Adam(lr=0.01, decay=1e-6)

#Compile model
model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy']
)

model.fit(
    train_x, train_y,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,#,
    verbose=1)

#Now I want to make sure that the we can predict the training set (using evaluate) and that it is the same result as during training
score = model.evaluate(train_x, train_y, batch_size=BATCH_SIZE, verbose=0)
print(' Train accuracy:', score[1])

The Output of the code is

Epoch 1/10 5872/5872 [==============================] - 0s 81us/sample - loss: 0.6954 - acc: 0.4997
Epoch 2/10 5872/5872 [==============================] - 0s 13us/sample - loss: 0.6924 - acc: 0.5229 
Epoch 3/10 5872/5872 [==============================] - 0s 14us/sample - loss: 0.6910 - acc: 0.5256
Epoch 4/10 5872/5872 [==============================] - 0s 13us/sample - loss: 0.6906 - acc: 0.5243 
Epoch 5/10 5872/5872 [==============================] - 0s 13us/sample - loss: 0.6908 - acc: 0.5238

Train accuracy: 0.52480716

So the problem is that the final modeling accuracy (0.5238) should be equal (evaluation) accuracy (0.52480716) which it is not. What have I done wrong here any help highly appreciated

Upvotes: 2

Views: 5380

Answers (2)

M B
M B

Reputation: 21

Thank you as shown below, it is not, which I dont understand

model = Sequential()
model.add(LSTM(32, batch_size=BATCH_SIZE, input_shape=(train_x.shape[1],train_x.shape[2]), return_sequences=True, stateful=False ))#,,return_sequences=Tru# stateful=True 
model.add(Dense(2, activation='softmax'))
opt = tf.keras.optimizers.Adam(lr=0.01, decay=1e-6)

#Compile model
model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy']
)

#Train model
model.fit(
    train_x, train_y,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    verbose=1,
    shuffle=False,
    validation_data=(validation_x, validation_y)]
)

score = model.evaluate(validation_x, validation_y, batch_size=BATCH_SIZE, verbose=0)

print(' Validation accuracy:', score[1])

Output

 Epoch 1/5 5872/5872 [==============================] - 3s 554us/sample - loss: 0.6923 - acc: 0.5154 - val_loss: 0.7149 - val_acc: 0.4668
 Epoch 2/5 5872/5872 [==============================] - 2s 406us/sample - loss: 0.6895 - acc: 0.4983 - val_loss: 0.7218 - val_acc: 0.4821
 Epoch 3/5 5872/5872 [==============================] - 2s 404us/sample - loss: 0.6890 - acc: 0.4940 - val_loss: 0.7230 - val_acc: 0.4821
 Epoch 4/5 5872/5872 [==============================] - 2s 406us/sample - loss: 0.6883 - acc: 0.4928 - val_loss: 0.7336 - val_acc: 0.4592
 Epoch 5/5 5872/5872 [==============================] - 2s 404us/sample - loss: 0.6881 - acc: 0.4934 - val_loss: 0.7278 - val_acc: 0.4745

 Validation accuracy: 0.45663264

Upvotes: 0

Manoj Mohan
Manoj Mohan

Reputation: 6054

"Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches."

https://keras.io/getting-started/faq/#why-is-the-training-loss-much-higher-than-the-testing-loss

For evaluation, since the trained model is taken, the accuracy is higher.

Upvotes: 0

Related Questions