CODEWITHSUNDEEP
pythonparametershyperopt
Minions
Minions

Reputation: 5487

Duplicated trials in Hyperopt library

I'm using hyperopt library to tune my model.

This is my search space:

search_space = {
            'max_df': hp.choice('max_df', [1, 0.95, 0.9]),
            'cls': hp.choice('cls', ['A', 'B', 'C', 'D', 'E', 'F', 'G',
                                     ]),
            'ngram_range': hp.choice('ngram_range', [
                (2,3), (2,4), (2,5), (2,6),
                (3,4), (3,5), (3,6),
                (4,5), (4,6), (5,6)
            ]),
        }

and This is my code:

trials = Trials()
best = fmin(self.objective_function, space=search_space, algo=tpe.suggest, max_evals=140, trials=trials)
bp = trials.best_trial['result']['Params']
print(bp)

Based on the number of possible parameter I have, the library should complete 210 iterates to finish the searching processes (3 * 7 * 10)

I set the parameter max_evals to 140 which is smaller than the possible total.

After each iterate I save the parameters I have with the score. What I found is that, even though I am searching in a lower space (140 instead of 210), there are trials (iterates) with duplicate parameters.

Does hyperopt library follow Gridsearch technique or it takes a random combination of parameters in each trial?

What I'm asking about is the parameter selection process, not the optimization technique (e.g. Bayesian optimization).

Upvotes: 3

Views: 815

Answers (1)

Max Pumperla
Max Pumperla

Reputation: 91

In your code you're using tpe (Tree-structured Parzen Estimator), which you can learn a bit more about in this paper by the author of hyperopt. I can't tell you too much here about this algorithm, but just note that each such search will start with a pre-defined "startup" period. Hyperopt by default uses 20 random trials to "seed" TPE, see here. Since your search space is fairly small and those random trials get picked independently, that already may account for your duplicates.

If you like, instead of TPE, you could also go with pure random search or a variant called ATPE in hyperopt.

Upvotes: 3

Related Questions