Reputation: 587
Should Test and Dev Set have different distribution from Training Set an Dev Set
Suppose I am building a network to localize an object. My training data consists of images captured at 5 different locations, and its a small data set ( each location has around 2k images ). Should I add all the images, shuffle them and then distribute them into training(60%), dev(20%), test(20%) or should I take data from 3 locations as Training, 1 location as Test and 1 location as dev.
Upvotes: 0
Views: 305
Answers (1)
Reputation: 101
Ideally the training set, test set and validation set should be drawn from the same distribution, so going by that you should add all the images, shuffle them and then distribute them into training(60%), dev(20%), test(20%). Also this would help you net be more invariant to location(as it would learn to ignore 5 locations against 3) and would have a better chance at generalization with the added diversity.
Upvotes: 1
Related Questions
- there is need to verify IID of dataset and perform statistical test for identical distribution after training and test data split?
- Using the dev set or the train set
- Does normalizing images by dividing by 255 leak information between train and test set?
- Is it a good idea to mix the validation / testing data with the training data?
- Same data split into training, dev and test set
- Classification task, result differ when test set distribution differ
- Can I reuse test data as training data?
- Mean error on training set equals mean error on testing set?
- How to randomly split a dataset into training set, test set, and dev set in Python?
- Why should my training set also be skewed in terms of number of class distribution just because my test set is skewed