CODEWITHSUNDEEP
machine-learningoptunity
ryuzakinho
ryuzakinho

Reputation: 1919

How to correctly evaluate a neural network model?

I am currently doing hyper-parameter optimization for my neural network.

I have train, dev and test file that were given to me. For my hyper-parameter optimization, I am running a complete training using the train and dev sets. In the end, I evaluate on the test set of the training for a given combination of parameters.

I am choosing the parameters that maximize the score on the test set. My issue is that I feel that it is wrong since I am sort of leaking the test set.

Is this procedure bad? Should I use optunity to maximize the accuracy on the dev set and in the end report a score on the test set?

Upvotes: 1

Views: 4783

Answers (1)

CanyonCat
CanyonCat

Reputation: 339

Typically the validation (dev) set is used to compare models with various hyper-parameters. Once your preferred model is chosen and trained, you run it on the test set to measure its performance.

Your intuition is correct; using the test set to chose model parameters is in a sense using that data to aid in the training procedure, which is not advisable.

The division and usage of train/validation/test sets are discussed in more detail in this post and in this video by Andrew Ng.

Upvotes: 2

Related Questions