Reputation: 1002
I have fitted a logistic regression model(using Smarket data set in ISLR package) using caret package in R. Then I calculated the total miss-classification error using (overall test error) K fold cross validation (K=5) as follows,
require(ISLR)
require(caret)
fitControl <- trainControl(method = "cv",number = 5)
mod_fit <- train(Direction ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume,
data=Smarket, method="glm",trControl = fitControlcv)
Generalized Linear Model
1250 samples
6 predictor
2 classes: 'Down', 'Up'
No pre-processing
Resampling: Leave-One-Out Cross-Validation
Summary of sample sizes: 1249, 1249, 1249, 1249, 1249, 1249, ...
Resampling results:
Accuracy Kappa
0.4976 -0.02588095
Form the above output i was able to calculate the total miss classification error because ,
total miss classification=1- Accuracy.
Is there any way to extract the sensitivity and specificity (class specific errors) also from the caret package using K fold cross validation ?
I was able to calculate sensitivity and specificity in K fold cross validation by creating using by creating a user defined function like mentioned here : https://youtu.be/AFg2MvhFeho
But i want to know whether that can be done easily using caret package .
Thank you
Upvotes: 1
Views: 1226
Reputation: 25
Have you tried using
confusionMatrix(data = predictions, reference = observations)
Should give you what you're looking for and more. You can also see more detail here.
Upvotes: 0
Reputation: 1002
It can be done by cross tabulating the observed and predicted values as follows,
table((mod_fit$pred)$obs,(mod_fit$pred)$pred)
Down Up
Down 125 477
Up 151 497
so
overall missclassification = (125+497)/250 = 0.4976
sensitivity = 497/(151+497) = 0.7770
Upvotes: 1