CODEWITHSUNDEEP
scikit-learngoogle-cloud-platformxgboostgoogle-cloud-ml
Francesco Cartella
Francesco Cartella

Reputation: 33

Hyperparameters tuning with Google Cloud ML Engine and XGBoost

I am trying to replicate the hyperparameter tuning example reported at this link but I want to use scikit learn XGBoost instead of tensorflow in my training application.

I am able to run multiple trials in a single job, on for each of the hyperparameters combination. However, the Training output object returned by ML-Engine does not include the finalMetric field, reporting metric information (see the differences in the picture below).

What I get with the example of the link above: Training output object with Tensorflow training app

What I get running my Training application with XGBoost: Training output object with XGBoost training app

Is there a way for XGBoost to return training metrics to ML-Engine?

It seems that this process is automated for tensorflow, as specified in the documentation:

How Cloud ML Engine gets your metric

You may notice that there are no instructions in this documentation for passing your hyperparameter metric to the Cloud ML Engine training service. That's because the service monitors TensorFlow summary events generated by your training application and retrieves the metric.

Is there a similar mechanism for XGBoost?

Now, I can always dump each metric results to a file at the end of each trial and then analyze them manually to select the best parameters. But, by doing so, am I loosing the automated mechanism offered by Cloud ML Engine, especially concerning the "ALGORITHM_UNSPECIFIED" hyperparameters search algorithm?

i.e.,

ALGORITHM_UNSPECIFIED: [...] applies Bayesian optimization to search the space of possible hyperparameter values, resulting in the most effective technique for your set of hyperparameters.

Upvotes: 0

Views: 1008

Answers (2)

tb.
tb.

Reputation: 789

Sara Robinson over at google put together a good post on how to do this. Rather than regurgitate and claim it as my own, I'll post this here for anyone else that comes across this post:

https://sararobinson.dev/2019/09/12/hyperparameter-tuning-xgboost.html

Upvotes: 0

lwz1992
lwz1992

Reputation: 304

Hyperparameter tuning support of XGBoost was implemented in a different way. We created the cloudml-hypertune python package to help do it. We're still working on the public doc for it. At the meantime, you can refer to this staging sample to learn about how to use it.

Upvotes: 1

Related Questions