CODEWITHSUNDEEP
rr-caretmlr
Brad
Brad

Reputation: 680

mlr equivalent of carets model selectionFunction in R

The caret library in R has a hyper-parameter 'selectionFunction' inside trainControl(). It's used to prevent over-fitting models using Breiman's one standard error rule, or tolerance, etc.

Does mlr have an equivalent? If so, which function is it within?

Upvotes: 1

Views: 155

Answers (2)

Brad
Brad

Reputation: 680

Posting an answer to my own question, I found this..

Estimate relative overfitting.

Source: R/relativeOverfitting.R

Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.

estimateRelativeOverfitting(
  predish,
  measures,
  task,
  learner = NULL,
  pred.train = NULL,
  iter = 1
)

Arguments

  • predish - (ResampleDesc ResamplePrediction Prediction) Resampling strategy or resampling prediction or test predictions.
  • measures - (Measure list of Measure) Performance measure(s) to evaluate. Default is the default measure for the task, see here getDefaultMeasure.
  • task - (Task) The task.
  • learner - (Learner character(1)) The learner. If you pass a string the learner will be created via makeLearner.
  • pred.train - (Prediction) Training predictions. Only needed if test predictions are passed.
  • iter - (integer) Iteration number. Default 1, usually you don't need to specify this. Only needed if test predictions are passed.

Upvotes: -1

pat-s
pat-s

Reputation: 6322

mlr only allows to choose optimal hyperparameters by optimizing certain measures/metrics.

However, essentially each "measure" in mlr is just a function that specifies how a certain performance is handled. You can try to write your own custom measure as outlined in this vignette.

Other than that, it might be worth opening this as a feature request in the new mlr3 framework, specifically in mlr3measures, since mlr itself is deprecated.

Upvotes: 2

Related Questions