CODEWITHSUNDEEP
lightgbmoptuna
masaya
masaya

Reputation: 490

Tuning num_iterations and learning_rate on LightGBM by Optuna

I would like to ask Optuna to tune num_iterations and learning_rate on LightGBM's Hyper-parameter for checking the balance between learning speed and accuracy.
Do you think the following code is appropriate and effective?

import lightgbm as lgb
from lightgbm import log_evaluation
from sklearn.model_selection import KFold

def objective(trial):

    param = {
        "objective": "binary",
        "metric": "binary_logloss",
        "boosting_type": "gbdt",                
        "seed": 42,
        'num_iterations': trial.suggest_int('num_iterations', 100, 10000),
        'learning_rate': trial.suggest_float('learning_rate',0.0001, 0.01 )
    }
    
    lgbcv = lgb.cv(param,
                   dtrain,
                   folds=KFold(n_splits=2)
                   verbose_eval=100,
                   callbacks=[lgb.early_stopping(stopping_rounds=250),
                              log_evaluation(period=100)]
                  )
    
    cv_score = lgbcv['binary_logloss-mean'][-1] + lgbcv['binary_logloss-stdv'][-1]
    return cv_score

study = optuna.create_study(direction='minimize')  
study.optimize(objective, n_trials=100)

As a general rule, if we reduce num_iterations, we should increase learning_rate. Since we know such a rule, it might not really necessary to have Optuna search for it. Even though, It might be still good to use Optuna to find out exactly what combination of values is good.

Anyway, If I can assume the combination I want to use, e.g., when num_iterations = 100 then learning_rate = 0.01 or when num_iterations = 1000 then learning_rate = 0.001 etc.,

Is there any way to limit some combinations and ask Optuna to do hyper-parameter search?

Upvotes: 0

Views: 1390

Answers (1)

OliverHennhoefer
OliverHennhoefer

Reputation: 979

You could replace the default univariate TPE sampler with the with the multivariate TPE sampler by just adding this single line to your code:

sampler = optuna.samplers.TPESampler(multivariate=True)

study = optuna.create_study(direction='minimize', sampler=sampler)  
study.optimize(objective, n_trials=100)

This sampler considers the dependencies of hyperparamters while optimization. You may suppress the occuring ExperimentalWarning like this:

warnings.filterwarnings("ignore", category=optuna.exceptions.ExperimentalWarning, module="optuna.*")

In the official documentation there is also the chapter "How can I optimize a model with some constraints?". I never used constraints, but maybe this can be adapted to your needs.

Upvotes: 2

Related Questions