Optuna suggest float log=True

Question

How can I have optuna suggest float numeric values from this list:

[1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1.0]

I'm using this Python code snippet:

trial.suggest_float("lambda", 1e-6, 1.0, log=True)

It correctly suggests values between 1e-6 and 1.0, but it suggests other values in the range, not just the values explicitly in the list above. What am I doing wrong?

Answer 1

What's wrong with suggest_categorical (in this instance)?

The suggest_categorical approach works alright, but it's subtly going to hurt the efficiency of your search in a way that can sometimes be extremely significant (especially if your list is large or you use this categorical approach many times in the same search space). Optuna considers each value in the list to be its own separate entity that cannot be ordered or compared to others. For example, I often use suggest_categorical to select between different algorithms, e.g. trial.suggest_categorical("algo", ["dbscan", "kmeans"]). But of course, it would be silly to say that dbscan > kmeans, or to find the value "halfway between" dbscan and kmeans. Optuna treats your list of numbers the same way. Even if it finds that performance is steadily and substantially decreasing as you move from 1.0 to 1e-5, it will still try 1e-6 because it cannot extrapolate on the trend. Note this is actually good, when your values are truly categorical and un-ordered, but bad in your case since we are trying parameters that are almost certainly bad.

What to do instead

trial.suggest_float selects between the min and max values provided in a continuous manner, unless the "step" argument is provided.

For example, trial.suggest_float("x", 0, 10) can return 0.0, 6.5, 3.25846, or anything else between 0 and 10. With step=0.5, it can only return numbers divisible by 0.5.

Sadly, the Optuna docs state:

The step and log arguments cannot be used at the same time. To set the step argument to a float number, set the log argument to False.

However, you can get around this. What I would suggest is having optima give you the exponent, and you calculate the actual value yourself. For example:

exp = trial.suggest_int("exp", -6, 0)
x = 10 ** exp

Essentially, anytime you can express the parameters you want as some function (in this case 10^x for x between -6 and 0), it is better to just code that function directly, with optima supplying the inputs, than try and bend optuna's functions too far to your purposes.

Answer 2

For selecting from a list, use suggest_categorical.

trial.suggest_categorical("lambda", [1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1.0])