CODEWITHSUNDEEP
rr-caret
slonopotam
slonopotam

Reputation: 1710

Selecting tuning parameters with caret using standard deviation of custom metric

I'm using caret with custom fitting metric, but I need to maximize not just this metric but lower bound of it's confidence interval. So I'd like to maximize something like mean(metric) - k * stddev(metric). I know how to do this manually, but is there a way to tell caret to automatically select best parameters using this function?

Upvotes: 4

Views: 5653

Answers (2)

Michael Kachala
Michael Kachala

Reputation: 1

There is more basic example in the caret's help for train function:

madSummary <- function (data,
                        lev = NULL,
                        model = NULL) {
  out <- mad(data$obs - data$pred, 
             na.rm = TRUE)  
  names(out) <- "MAD"
  out
}

robustControl <- trainControl(summaryFunction = madSummary)
marsGrid <- expand.grid(degree = 1, nprune = (1:10) * 2)

earthFit <- train(medv ~ .,
                  data = BostonHousing, 
                  method = "earth",
                  tuneGrid = marsGrid,
                  metric = "MAD",
                  maximize = FALSE,
                  trControl = robustControl)

Upvotes: 0

David
David

Reputation: 9405

Yes, you can define your own selection metric through the "summaryFunction" parameter of your "trainControl" object and then with the "metric" parameter of your call to train(). Details on this are pretty well documented in the "Alternate Performance Metrics" section on caret's model tuning page: http://caret.r-forge.r-project.org/training.html

I don't think you gave enough information for anyone to write exactly what you're looking for, but here is an example using the code from the twoClassSummary function:

> library(caret)
> data(Titanic)
> 
> #an example custom function
> roc <- function (data, lev = NULL, model = NULL) {
+   require(pROC)
+   if (!all(levels(data[, "pred"]) == levels(data[, "obs"]))) 
+     stop("levels of observed and predicted data do not match")
+   rocObject <- try(pROC:::roc(data$obs, data[, lev[1]]), silent = TRUE)
+   rocAUC <- if (class(rocObject)[1] == "try-error") 
+     NA
+   else rocObject$auc
+   out <- c(rocAUC, sensitivity(data[, "pred"], data[, "obs"], lev[1]), specificity(data[, "pred"], data[, "obs"], lev[2]))
+   names(out) <- c("ROC", "Sens", "Spec")
+   out
+ }
> 
> #your train control specs
> tc <- trainControl(method="cv",classProb=TRUE,summaryFunction=roc)
> #yoru model with selection metric specificed
> train(Survived~.,data=data.frame(Titanic),method="rf",trControl=tc,metric="ROC")
32 samples
 4 predictors
 2 classes: 'No', 'Yes' 

No pre-processing
Resampling: Cross-Validation (10 fold) 

Summary of sample sizes: 28, 29, 30, 30, 28, 28, ... 

Resampling results across tuning parameters:

  mtry  ROC    Sens  Spec  ROC SD  Sens SD  Spec SD
  2     0.9    0.2   0.25  0.175   0.35     0.425  
  4     0.85   0.4   0.6   0.211   0.459    0.459  
  6     0.875  0.35  0.6   0.212   0.412    0.459  

ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 2.

Upvotes: 6

Related Questions