CODEWITHSUNDEEP
pythonrxgboost
mathsnerd
mathsnerd

Reputation: 103

Custom Error Metric not changing predictions XGBoost R

I have created a custom error metric which prints as I run the XGBoost xgb.train but does not actually have any affect on the output. From what I can tell it is simply printing the custom error metric for the round but not using that to determine the accuracy.

I think this because the prediction outputs is exactly the same to when I use the default error metric. I have also tried hard coding the error output to a static 1 so that the output should be random but the result was exactly the same.

Do I need to create a custome objective function for the custom error metric to work?

Thanks!

My code: 

# xgboost fitting with arbitrary parameters
xgb_params_1 = list(
  objective = "reg:linear",                                               
  eta = 0.2,                                                       
  max.depth = 6,                                                      
  booster =  "gbtree"   
)


evalerror <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")


  score <- as.numeric((sum(preds[1:1000]) - sum(labels[1:1000] )) / sum(labels[1:1000]) ) 

  return(list(metric="custom_error",value=1))

}


myWatch <- list(val=dvalid,train=dtrain)
# fit the model with the arbitrary parameters specified above
xgb_1 = xgb.train(data = dtrain,
                params = xgb_params_1,
                nrounds = 150,                                            
                nthread = 6,
                verbose = T,                                         
                print_every_n = 50,
                watchlist = myWatch,
                early_stop_round = 1000,                                                    
                eval_metric = evalerror,
                disable_default_eval_metric = 1

)

# Perform a prediction
pred <- predict(xgb_1, dvalid)
results <- cbind(as.data.table(pred), as.data.table(data[year > trainEndDate,"total_installs"]))


#Compute test RMSE
sqrt(mean((results$pred - results$total_installs)**2))

Printed error metrics:

enter image description here

Upvotes: 0

Views: 1218

Answers (1)

astro_asz
astro_asz

Reputation: 2318

Custom eval_metric is just for evaluation purposes. It is displayed at every round (when using watches) and it is useful to tune number of boosting rounds, and you can use it when you do cross-validation to tune your parameters to maximise/minimise your metric. I use it in particular to tune my learning rate to make the model converge faster with less rounds.

Custom objective function is a completely different beast and it is not the same as evaluation metric. It is more of a type of model like classification, regression etc. It drives the convergence of the model. If you still want it here is an example of xgboost regression objective.

Upvotes: 1

Related Questions