CODEWITHSUNDEEP
pythonmachine-learningneural-networkcaffeconv-neural-network
mjacuse
mjacuse

Reputation: 79

Caffe: train, validation and test split

I've been using caffe for a while, with some success, but I have noticed in examples given that there is only ever a two way split on the data set with TRAIN and TEST phases, where the TEST set seems to act as a validation set.

Ideally I would like to have three sets, so that once the model is trained, I can save it and test it on a completely new test set - stored in a completed separate lmdb folder. Does anyone have any experience of this? Thanks.

Upvotes: 2

Views: 4126

Answers (1)

ypx
ypx

Reputation: 1459

Differentiating between validation and testing is made to imply that hyperparameters may be tuned to the validation set while nothing is fitted to the test set in any way. caffe doesn't optimize anything but the weights, and since the test is only there for evaluation, it does exactly as expected.

Assuming you're tuning hyper parameters between solver optimization runs. The lmdb passed to caffe for testing is really the validation set. If you're done with tuning your hyperparameters and do one more solver optimization with an lmdb for testing that holds data never used in previous runs. That last lmdb is your test set.

Since caffe doesn't optimize hyperparameters, its test set is what it is, a test set. It's possible to come up with a some python code around the solver optimization calls that iterates through hyperparameter values. After it's done it can swap in a new lmdb with unseen data to tell you about how well the network generalizes with it.

I don't recommend modifying caffe for an explicit val/test distinction. You don't even have to do anything elaborate with setting up the prototxt file for the solver and network definition. You can do the val/test swap at the end by simply moving the val lmdb somewhere else and moving the test lmdb in its place using shutil.copy(src, dst)

Upvotes: 5

Related Questions