CODEWITHSUNDEEP
matlabsvmlibsvmcross-validation
TestGuest
TestGuest

Reputation: 603

LibSVM - what to do after crossvalidation?

I use a grid-search for finding optimal parameters C and gamma for a radial basis SVM with the code below (code largely taken from Retraining after Cross Validation with libsvm). This works fine, but I have two upcoming questions:

1.Given I have just one dataset at disposal - what do I do with the optimal values of C and gamma? Do I split up my dataset and use one part just for determining the optimal parameters C and gamma with gridsearch, and use the second part then to predict accuracy with these parameters?

2.Given I have another dataset after having found optimal C and gamma for my old dataset - why should I use these C and gamma for the new dataset, instead of applying gridsearch to the new dataset too, to find its new optimal parameters?

Thanks

%# read example data from libsvm
[labels,data] = libsvmread('./heart_scale');

%# grid of parameters
folds = 5;

[C,gamma] = meshgrid(-5:2:15, -15:2:3);

%# grid search, and cross-validation
cv_acc = zeros(numel(C),1);
for i=1:numel(C)
    cv_acc(i) = svmtrain(labels, data, ...
                    sprintf('-c %f -g %f -v %d', 1^C(i), 2^gamma(i), folds));
end

%# pair (C,gamma) with best accuracy
[~,idx] = max(cv_acc);


%# now you can train you model using best_C and best_gamma
best_C = 2^C(idx);
best_gamma = 2^gamma(idx);

Upvotes: 0

Views: 167

Answers (2)

ABC
ABC

Reputation: 665

Once the optimal parameters are there you assume that they are the optimal for all data everywhere that belongs to this class. Of course this is a grand assumption, and having a larger dataset would decrease the variability of the change in these parameters. But the whole point of learning is given a piece of brand new data can I guess what it is, or what it means? So training again on new data defeats the purpose of seeing what would happen on a piece of data you've never seen.

Now in my experience these parameters only optimize the performance within a certain ball park. That is to say, it can take your test set accuracy from say 80% to 81-82%. The more important thing in most cases is getting to the ball park of 80% which depends on designing your system well, picking the correct features and preprocessing these features. Then once this is done, work on upping your performance by tweeking these parameters.

Anyways, hopes this helps.

Upvotes: 1

user1809923
user1809923

Reputation: 1255

You conduct a gridsearch to find the optimal parameters for your SVM for a given problem. You do this on a dataset that is available to you and that includes annotations, i.e., where you know what predictions are right and wrong - how else would you find out which values for C and gamma are the best ones?

Once obtained, you assume that your previously used dataset reflected the problem well enough and from here on after you can use your SVM with the values for C and gamma on a dataset for the same problem that does not have annotations.

Upvotes: 1

Related Questions