CODEWITHSUNDEEP
rmachine-learningr-caretrocauc
Chris
Chris

Reputation: 2071

Evaluate ROC metric, caret package - R

I have this code:

model_nn <- train(
  Y ~ ., training,
  method = "nnet",
  metric = "ROC",
  trControl = trainControl(
    method = "cv", 
    number = 10,
    verboseIter = TRUE,
    classProbs = TRUE,
    summaryFunction = twoClassSummary
  )
)

nnprediction <- predict(model_nn, testing)
cmnn <-confusionMatrix(nnprediction,testing$Y)
print(cmnn)

Which works. However, I can't evaluate how well is the ROC metric performance with the confusionMatrix command. How can I evaluate it, in order to try a different set of variables and/or machine learning algorithms to improve the ROC performance?

PS: The dependent variable is a factor of two classes.

Upvotes: 0

Views: 2436

Answers (1)

desertnaut
desertnaut

Reputation: 60319

Just typing model_nn will give you the AUC score for the different settings used during training; here is an example, using the first 100 records of the iris data (2 classes):

library(caret)
library(nnet)

data(iris)
iris_reduced <- iris[1:100,]
iris_reduced <- droplevels(iris_reduced, "virginica")

model_nn <- train(
  Species ~ ., iris_reduced,
  method = "nnet",
  metric = "ROC",
  trControl = trainControl(
    method = "cv", 
    number = 5,
    verboseIter = TRUE,
    classProbs = TRUE,
    summaryFunction = twoClassSummary
  )
)

model_nn

Result:

Neural Network 

100 samples
  4 predictors
  2 classes: 'setosa', 'versicolor' 

No pre-processing
Resampling: Cross-Validated (5 fold) 
Summary of sample sizes: 80, 80, 80, 80, 80 
Resampling results across tuning parameters:

  size  decay  ROC  Sens  Spec
  1     0e+00  1.0  1.0   1   
  1     1e-04  0.8  0.8   1   
  1     1e-01  1.0  1.0   1   
  3     0e+00  1.0  1.0   1   
  3     1e-04  1.0  1.0   1   
  3     1e-01  1.0  1.0   1   
  5     0e+00  1.0  1.0   1   
  5     1e-04  1.0  1.0   1   
  5     1e-01  1.0  1.0   1   

ROC was used to select the optimal model using  the largest value.
The final values used for the model were size = 1 and decay = 0.1.

BTW, the term "ROC" here is somewhat misleading: what is returned is not of course the ROC (which is a curve, and not a number), but the area under the ROC curve, i.e. the AUC (using metric='AUC' in trainControl has the same effect).

Upvotes: 2

Related Questions