CODEWITHSUNDEEP
neural-networkkeras
Stephen Johnson
Stephen Johnson

Reputation: 517

Test score vs test accuracy when evaluating model using Keras

Im using a neural network implemented with the Keras library and below is the results during training. At the end it prints a test score and a test accuracy. I can't figure out exactly what the score represents, but the accuracy I assume to be the number of predictions that was correct when running the test.

Epoch 1/15 1200/1200 [==============================] - 4s - loss: 0.6815 - acc: 0.5550 - val_loss: 0.6120 - val_acc: 0.7525

Epoch 2/15 1200/1200 [==============================] - 3s - loss: 0.5481 - acc: 0.7250 - val_loss: 0.4645 - val_acc: 0.8025

Epoch 3/15 1200/1200 [==============================] - 3s - loss: 0.5078 - acc: 0.7558 - val_loss: 0.4354 - val_acc: 0.7975

Epoch 4/15 1200/1200 [==============================] - 3s - loss: 0.4603 - acc: 0.7875 - val_loss: 0.3978 - val_acc: 0.8350

Epoch 5/15 1200/1200 [==============================] - 3s - loss: 0.4367 - acc: 0.7992 - val_loss: 0.3809 - val_acc: 0.8300

Epoch 6/15 1200/1200 [==============================] - 3s - loss: 0.4276 - acc: 0.8017 - val_loss: 0.3884 - val_acc: 0.8350

Epoch 7/15 1200/1200 [==============================] - 3s - loss: 0.3975 - acc: 0.8167 - val_loss: 0.3666 - val_acc: 0.8400

Epoch 8/15 1200/1200 [==============================] - 3s - loss: 0.3916 - acc: 0.8183 - val_loss: 0.3753 - val_acc: 0.8450

Epoch 9/15 1200/1200 [==============================] - 3s - loss: 0.3814 - acc: 0.8233 - val_loss: 0.3505 - val_acc: 0.8475

Epoch 10/15 1200/1200 [==============================] - 3s - loss: 0.3842 - acc: 0.8342 - val_loss: 0.3672 - val_acc: 0.8450

Epoch 11/15 1200/1200 [==============================] - 3s - loss: 0.3674 - acc: 0.8375 - val_loss: 0.3383 - val_acc: 0.8525

Epoch 12/15 1200/1200 [==============================] - 3s - loss: 0.3624 - acc: 0.8367 - val_loss: 0.3423 - val_acc: 0.8650

Epoch 13/15 1200/1200 [==============================] - 3s - loss: 0.3497 - acc: 0.8475 - val_loss: 0.3069 - val_acc: 0.8825

Epoch 14/15 1200/1200 [==============================] - 3s - loss: 0.3406 - acc: 0.8500 - val_loss: 0.2993 - val_acc: 0.8775

Epoch 15/15 1200/1200 [==============================] - 3s - loss: 0.3252 - acc: 0.8600 - val_loss: 0.2960 - val_acc: 0.8775

400/400 [==============================] - 0s

Test score: 0.299598811865

Test accuracy: 0.88

Looking at the Keras documentation, I still don't understand what score is. For the evaluate function, it says:

Returns the loss value & metrics values for the model in test mode.

One thing I noticed is that when the test accuracy is lower, the score is higher, and when accuracy is higher, the score is lower.

Upvotes: 26

Views: 51844

Answers (2)

maz
maz

Reputation: 1980

For reference, the two relevant parts of the code:

model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

score, acc = model.evaluate(x_test, y_test,
                                batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

Score is the evaluation of the loss function for a given input.

Training a network is finding parameters that minimize a loss function (or cost function).

The cost function here is the binary_crossentropy.

For a target T and a network output O, the binary crossentropy can defined as: 1

f(T,O) = -(T * log(O) + (1-T) * log(1-O))

So the score you see is the evaluation of that.

If you feed it a batch of inputs it will most likely return the mean loss. 2

So yeah, if your model has lower loss (at test time), it should often have lower prediction error.

1 See similar formula at BCELoss on PyTorch (Binary Cross Entropy = BCE)

2 Note how the PyTorch BCE is reduced to a scalar using the default value of reduction ='mean' (i.e. average) by default

Upvotes: 32

Praveen
Praveen

Reputation: 2177

Loss is often used in the training process to find the "best" parameter values for your model (e.g. weights in neural network). It is what you try to optimize in the training by updating weights.

Accuracy is more from an applied perspective. Once you find the optimized parameters above, you use this metrics to evaluate how accurate your model's prediction is compared to the true data.

This answer provides a detailed info:

How to interpret "loss" and "accuracy" for a machine learning model

Upvotes: 5

Related Questions