CODEWITHSUNDEEP
rneural-network
Sam
Sam

Reputation: 654

Calculating AUC from nnet model

For a bit of background, I am using the nnet package building a simple neural network.

My dataset has a number of factor and continuous variable features. To handle the continuous variables I apply scale and center which minuses each by its mean and divides by its SD.

I'm trying to produce an ROC & AUC plot from the results of neural network model.

The below is the code used to build my basic neural network model: 

model1 <- nnet(Cohort ~ .-Cohort, 
           data = train.sample,
           size = 1)

To get some predictions, I call the following function: 

train.predictions <- predict(model1, train.sample)

Now, this assigns the train.predictions object to a large matrix consisting of 0 & 1 values. What I want to do, is getting the class probabilities for each prediction so I can plot an ROC curve using the pROC package.

So, I tried adding the following parameter to my predict function: 

train.predictions <- predict(model1, train.sample, type="prob")

But I get an error:

Error in match.arg(type) : 'arg' should be one of “raw”, “class”

How can I go about getting class probabilities from outputs?

Upvotes: 2

Views: 2817

Answers (1)

Mitch Wheat
Mitch Wheat

Reputation: 300827

Assuming your test/validation data set is in train.test, and train.labels contains the true class labels:

train.predictions <- predict(model1, train.test, type="raw")

## This might not be necessary:
detach(package:nnet,unload = T)

library(ROCR)

## train.labels:= A vector, matrix, list, or data frame containing the true  
## class labels. Must have the same dimensions as 'predictions'.

## computing a simple ROC curve (x-axis: fpr, y-axis: tpr)
pred = prediction(train.predictions, train.labels)
perf = performance(pred, "tpr", "fpr")
plot(perf, lwd=2, col="blue", main="ROC - Title")
abline(a=0, b=1)

Upvotes: 3

Related Questions