How can I get Gini index after doing a grid search in hyper parameter tuning of GBM for a tweedie loss function?

Question

I am doing hyperparameter tuning for gbm model in H2o and since my loss function is Tweedie I don't want to look at mse as my model selection criteria.

In H2o documentation, it says that Gini index can be calculated for both regression and classification models, however when I try to get it for my Tweedie regression model, it returns null. Below is how I get the best model and score it on the test set.

gbm_sorted_grid <- h2o.getGrid(grid_id = "grid_hp4", sort_by = 
"residual_deviance")

best_model <- h2o.getModel(gbm_sorted_grid@model_ids[[1]])  

perf <- h2o.performance(best_model, newdata = lrs_test)
h2o.giniCoef(perf)
Null

And when I try the code below, I get below error:

h2o.giniCoef(best_model) : No Gini for H2ORegressionModel

Does this only work on Bernoulli distribution?

Answer 1

The giniCoef in H2O-3 only supports binary classification problems, this is a mistake in the documentation. I've created a jira ticket so that the user guide gets updated. Thanks for highlighting the issue!