CODEWITHSUNDEEP
rmachine-learningclassificationroc
AdeeThyag
AdeeThyag

Reputation: 125

Unable to get the proper multi-class ROC Curve using the pROC library in R

My prediction column has spam, not spam and undefinable. I used the ensemble method-stacking method to predict it.I was able to get an accuracy of around 77% I am able to plot the ROC Curve but I just don't think it is right.

Ensemle Technique code:

# Generate level-one dataset for training the ensemble metalearner
predDF <- data.frame(dataTest.pred, NB_Predictions, RF_Predictions,SVM_Predictions, spam = validation$spam, stringsAsFactors = F)

# Train the ensemble

# define training control
set.seed(1958)
# Train the ensemble
modelStack <- caret::train(spam ~ ., data = predDF, method = "rf")

Here is my code for the ROC:

#ROC Curve
pre<-predict(modelStack, testPredLevelOne,type='prob')
# AUC measure
modelroc = mean(
  c(as.numeric(multiclass.roc(testPredLevelOne$spam, pre[,1])$auc),
    as.numeric(multiclass.roc(testPredLevelOne$spam, pre[,2])$auc),
    as.numeric(multiclass.roc(testPredLevelOne$spam, pre[,3])$auc)
  )
)

I don't think the AUC measure is also right.

> modelroc
[1] 0.500903

Here is another method that I tried to plot the ROC curve. I don't think that is working either.

#Method 2 ROC
predictions<-as.numeric(predict(modelStack, testPredLevelOne))
roc.multi<-multiclass.roc(testPredLevelOne$spam,predictions)
auc(roc.multi)
rs <- roc.multi[['rocs']]
plot.roc(rs[[1]])

sapply(2:length(rs),function(i) lines.roc(rs[[i]],col=i))

Here is the Plot:

enter image description here

Could someone please help me out with this? Thanks a lot.

Upvotes: 1

Views: 2701

Answers (1)

Calimo
Calimo

Reputation: 7969

The averaging between the classes is done directly within pROC. Therefore you want to run multiclass.roc only once, with a single vector of predictions. Typically you want to use type="response" instead of probs, although this may vary depending on the predict function available for your model:

pre<-predict(modelStack, testPredLevelOne, type='response')

After that, pROC does the averaging for you so you can get the average AUC directly:

multiclass.roc(testPredLevelOne$spam, pre)$auc

Note that in your case this will be the average over 3 AUCs: spam vs. not spam, spam vs. undefinable and not spam vs. undefinable. This may not reflect the value of accuracy you calculated.

Regarding the plot, you obtained the predicted class from the predict.train method from caret. Usually in order to construct a ROC curve, you want a numeric, quantitative measurement. A qualitative measurement results in a ROC curve containing a single point, which is usually not optimal. Unfortunately the type="probs" option from caret returns 3 vectors of probabilities, which is not a format supported by pROC.

An alternative would be to do perform the calculation more manually, selecting the correct probability column and levels you want to test:

pre<-predict(modelStack, testPredLevelOne,type='prob')
roc(testPredLevelOne$spam, pre[,X], levels = c("not spam", "spam"), plot = TRUE)
roc(testPredLevelOne$spam, pre[,X], levels = c("undefined", " spam"), plot = TRUE, add = TRUE)

Ultimately, you should carefully review the relevance of multiclass ROC analysis. ROC was designed for binary classification, and the relevance of the various existing multiclass extensions is somewhat dubious in my experience.

Upvotes: 3

Related Questions