CODEWITHSUNDEEP
rglm
NR117
NR117

Reputation: 25

Is and Brier Score the right approach to evaluate this binomial glm model?

I want to evaluate a calculator that predicts events by calculating an estimated percentage of the probability that the event will occur based on numerous input variables (in practice it's about patients with risk factors and postoperative complications). Unfortunately I have no insight on how the calculator actually works but can only look at input/output values. After entering a number of scenarios I end up with a df like this:

risk <- runif(100, 0, 1)
event <- sample(c(0,1), 100, replace = TRUE) 
test.df <- data.frame(risk, event)

Risk describes the predicted probability and event describes if the complication actually occurred.

Now I want to find out how good the calculator is at predicting complications in my patients.

Would it be suitable to use a binomial glm and evaluate it by using a scaled brier score?

library (stats)
library (DescTools)
mod <- glm(event ~ risk, family = binomial, data = test.df)
BrierScore(mod, scaled = TRUE)

From my perspective one could also directly calculate a Brier Score and Scaled Brier Score from the df, but I'm unsure if this would be the correct approach.

BS <- function(obs, pred) {
          mean((obs - pred)^2)
        }
        
Scaled_BS <- function(obs, pred) {
          1 - (BS(obs, pred) / (mean(obs) * (1 - mean(obs))))
        }
BS(event, risk)
Scaled_BS(event, risk)

Please forgive me if this is a completely wrong approach. I'm new to this kind of task and any help would be much appreciated!

Upvotes: 0

Views: 233

Answers (1)

Duck
Duck

Reputation: 39613

Using the Brier score is fine but you can complement with ROC analysis which is designed to test binary outcomes. The key in ROC analysis is returning a value between 0-1. The closest to 1 is, the better the predictions are. And if the value is close to 0.5 then the prediction is random and lack of predictive power. The rule of thumb is that the performance for a model is proper if the AUROC is over 0.7. I have sketched a code for you using ROCR package:

library(ROCR)
#Data
risk <- runif(100, 0, 1)
event <- sample(c(0,1), 100, replace = TRUE) 
test.df <- data.frame(risk, event)
#Model
mod <- glm(event ~ risk, family = binomial, data = test.df)
#Evaluate
test.df$score<-predict(mod,type='response',test.df)
pred<-prediction(test.df$score,test.df$event)
discr=performance(pred,"auc")
#Compute AUROC
AUROC=as.numeric([email protected])

The output for AUROC is:

AUROC
[1] 0.5700522

Which is low. Also you can plot the ROC curve, the closest the curve is to left superior corner means that predictions come from a high performance model. In the case of your data:

enter image description here

Which is the middle of the square and not close to the left superior corner. This measure can help you to determine the performance of your model.

Upvotes: 1

Related Questions