CODEWITHSUNDEEP
rcross-validationr-caret
student_R123
student_R123

Reputation: 1002

Regarding the Caret package in R when apply K fold cross validation

I have fitted a logistic regression model(using Smarket data set in ISLR package) using caret package in R. Then I calculated the total miss-classification error using (overall test error) K fold cross validation (K=5) as follows,

require(ISLR)
require(caret)
fitControl <- trainControl(method = "cv",number = 5)
mod_fit <- train(Direction ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume,
                 data=Smarket, method="glm",trControl = fitControlcv)

Generalized Linear Model 

1250 samples
   6 predictor
   2 classes: 'Down', 'Up' 

No pre-processing
Resampling: Leave-One-Out Cross-Validation 
Summary of sample sizes: 1249, 1249, 1249, 1249, 1249, 1249, ... 
Resampling results:

  Accuracy  Kappa      
  0.4976    -0.02588095

Form the above output i was able to calculate the total miss classification error because ,

total miss classification=1- Accuracy.

Is there any way to extract the sensitivity and specificity (class specific errors) also from the caret package using K fold cross validation ?

I was able to calculate sensitivity and specificity in K fold cross validation by creating using by creating a user defined function like mentioned here : https://youtu.be/AFg2MvhFeho

But i want to know whether that can be done easily using caret package .

Thank you

Upvotes: 1

Views: 1226

Answers (2)

corio
corio

Reputation: 25

Have you tried using 

confusionMatrix(data = predictions, reference = observations)

Should give you what you're looking for and more. You can also see more detail here.

Upvotes: 0

student_R123
student_R123

Reputation: 1002

It can be done by cross tabulating the observed and predicted values as follows,

table((mod_fit$pred)$obs,(mod_fit$pred)$pred)
      Down  Up
  Down  125 477
  Up    151 497

so 

overall missclassification = (125+497)/250 = 0.4976

sensitivity =  497/(151+497) = 0.7770

Upvotes: 1

Related Questions