Reputation: 680
The caret library in R has a hyper-parameter 'selectionFunction' inside trainControl(). It's used to prevent over-fitting models using Breiman's one standard error rule, or tolerance, etc.
Does mlr have an equivalent? If so, which function is it within?
Upvotes: 1
Views: 155
Reputation: 680
Posting an answer to my own question, I found this..
Source: R/relativeOverfitting.R
Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.
estimateRelativeOverfitting(
predish,
measures,
task,
learner = NULL,
pred.train = NULL,
iter = 1
)
Arguments
character(1)
) The learner. If you pass a string the learner will be created via makeLearner. Upvotes: -1
Reputation: 6322
mlr only allows to choose optimal hyperparameters by optimizing certain measures/metrics.
However, essentially each "measure" in mlr is just a function that specifies how a certain performance is handled. You can try to write your own custom measure as outlined in this vignette.
Other than that, it might be worth opening this as a feature request in the new mlr3 framework, specifically in mlr3measures, since mlr itself is deprecated.
Upvotes: 2