How are val_loss and val_accuracy implemented in tensorflow / Keras?

Question

I would like to add logs to my training similar to val_loss and val_accuracy. Specifically, I have another dataset (not training or validation) which I would like to track model evaluation statistics of during training. I would like to do this in the best possible alignment with the design of Keras.

I haven't been able to see how the built-in val_loss and val_accuracy are implemented. The ways that seem obvious to me are callbacks and metrics. However, I have checked the built-in callbacks and do not find one similar, and unless I am mistaken metrics have a restricted intent and are not intended for general operations such as additional model evaluations. It seems straightforward to implement with a callback, but I would prefer to base it on an existing implementation so I can be aware of any best practices. I assume that if the framework authors chose not to just use a callback for this there will be a reason. Could you please let me know where I can find the implementation of val_loss and val_accuracy?

Answer 1

I have found that validation statistics are currently implemented in a bespoke way https://github.com/tensorflow/tensorflow/blob/v2.4.1/tensorflow/python/keras/engine/training.py#L1113-L1143 . This means unfortunately there is no apparent intended way for adding additional similar evaluations.