CODEWITHSUNDEEP
python-3.x
Uwe.Schneider
Uwe.Schneider

Reputation: 1415

Using Bayesian Optimization with discrete grid points

I am using the BayesianOptimization package to optimize Hyperparameters of a sklearn Decision tree. There are continuous parameters like

min_weight_fraction_leaf in the intervall (0.1,0.4)

but also discrete parameters like

criterion = "mse","friedman_mse",...

or combinations of None and ints like

max_depth = None, 1,2,...

def DT_optimizer_function(criterion,max_depth,min_weight_fraction_leaf):
    """
    Function with unknown internals we wish to maximize.
    """
    return -x ** 2 - (y - 1) ** 2 + 1

from bayes_opt import BayesianOptimization

# Bounded region of parameter space
pbounds = {'criterion': ?,
           'max_depth': ?,
           'min_weight_fraction_leaf' = (0.1,0.4)
          }

optimizer = BayesianOptimization(
    f=DT_optimizer_function,
    pbounds=pbounds,
    random_state=1,
)

optimizer.maximize(
    init_points=2,
    n_iter=3,
)

print(optimizer.max)

Is it possible to optimize the discrete values too?

Upvotes: 4

Views: 1758

Answers (1)

Zahra
Zahra

Reputation: 7197

According to their docs

There is no principled way of dealing with discrete parameters using this package.

However, you may cast the floating point sample from bayes opt to int or process it in a way that serves your purpose. For example, in your snippet above you may do the following:

from bayes_opt import BayesianOptimization

def function_to_be_optimized(criterion,max_depth):
   # map bayes_opt sample to a categorical value
   # values below 0.5 mapped to 'mse', above 0.5 mapped to 'friedman_mse' 
   criterion_list = ['mse', 'friedman_mse'] 
   criterion = criterion_list[round(criterion)] 
   
   # map bayes_opt sample to an integer value
   max_depth = int(max_depth)
   max_depth = max_depth if max_depth!=0 else None

   magical_number = ...
   return magical_number
   

pbounds = {
           'criterion': (0,1),
           'max_depth': (0,5)
          }

optimizer = BayesianOptimization(
    f=function_to_be_optimized,
    pbounds=pbounds,
    random_state=42,
)

optimizer.maximize(
    init_points=2,
    n_iter=3,
)

I understand that this is not an ideal solution since the categorical values may not enjoy the smoothness properties that the a real valued parameter does.

Upvotes: 1

Related Questions