CODEWITHSUNDEEP
machine-learningkerasdeep-learning
John M.
John M.

Reputation: 2712

Keras: How come 'accuracy' is higher than 'val_acc'?

I've compiled a model with metrics=['accuracy'] and the value I get is always higher than validation accuracy val_acc. For instance:

Epoch 19/20
53/53 [==============================] - 280s - loss: 0.3931 - acc: 0.8238 - val_loss: 0.5002 - val_acc: 0.7757
Epoch 20/20
53/53 [==============================] - 278s - loss: 0.3957 - acc: 0.8255 - val_loss: 0.5009 - val_acc: 0.7754
accuracy: 0.790697674418604

Anyone know how the two values are computed differently?

UPDATE

I'm on Keras 2.0.8. By 'accuracy', I'm referring to the last line accuracy: 0.790697674418604. I didn't provide it with a testing set, so I wonder what it tests on.

Upvotes: 9

Views: 32323

Answers (3)

Varsha Venkatesh
Varsha Venkatesh

Reputation: 320

The validation accuracy is computed on a dataset that is separate from your training set. The validation accuracy can be lower for multiple reasons - I'd look at imbalanced classes, overfitting, not enough data if there is a large discrepancy between the training set. The accuracy you cite is likely the training accuracy.As to why training accuracy is being calculated at all - you feed your input aka features into the network and initialize the NN with random weights. You let the NN figure out/guess the label and then compare it with the actual label. The error.i.e. the difference between the guess and the actual label is computed and backpropagate the error. You do this for all samples/batches. Keras handles all of this automatically and outputs metrics for how the training is proceeding - usually loss and the accuracy as the samples/batches are added up. That's the accuracy metric you see.

Upvotes: 3

ucohen1
ucohen1

Reputation: 5

The actual metric the model is optimizing for is the loss, so during the optimization process you should follow the loss and monitor its improvements compared to the val_loss. At the point of val_loss getting further than the loss you are over-fitting and need to take adequate measures.

The accuracy measure greatly depends on your data as it counts for the percentage of correct classifications. Imbalanced classes in your data splits could lead to different accuracy measures.

Upvotes: 0

user007
user007

Reputation: 548

During training, the samples are again split into 2 internal subsets. One which is used for actual training and other which is used for validation after each epoch. The ratio of split can be controlled by the parameter 'validation_split' like below (Example from Keras)

h = model.fit(X_train, Y_train, batch_size=200, epochs=50, verbose=2, validation_split=0.2)

Now, coming to the log, 'acc' refers to accuracy of what was trained against. 'val_acc' refers to validation set. Note that val_acc refers to a set of samples that was not shown to the network during training and hence refers to how much your model works in general for cases outside the training set.

It is common for validation accuracy to be lower than accuracy. But ideally, you should strive to keep those values at the same level. If validation accuracy is much lower than accuracy, you are certainly over fitting (like in the example above) - where the accuracy is 84 and validation accuracy is 77.

Edit: Regarding 'accuracy' in the last line of the log, that is the accuracy of the network after all epochs run against the test data set. This is usually more closer to 'val_acc' than accuracy (like in the case above it is 79). This just means that the samples in test data alinged more closely than samples in validation data in the last epoch run (remember both these sets are not used for training)

In any case, I think you should tweak to make sure that 'acc' and 'val_acc' and final 'accuracy' are more closer to each other

Upvotes: 12

Related Questions