CODEWITHSUNDEEP
pythonscikit-learnhyperparameters
kathryn
kathryn

Reputation: 33

Conditional tuning of hyperparameters with RandomizedSearchCV in scikit-learn

I want to use RandomizedSearchCV in sklearn to search for the optimal hyperparameter values for a support vector classifier on my dataset. The hyperparameters I am optimising are "kernel", "C" and "gamma". However, in the case of a "poly" kernel, I would also like to optimise a fourth hyperparameter, "degree" (the index of the polynomial kernel function).

I realise that since the degree hyperparameter is ignored when the kernel is not "poly", I could just include degree in the params dictionary I provide to RandomizedSearchCV (as I've done in the code below). However, ideally I would like to search uniformely across the non-poly kernels plus each degree of poly kernel, i.e. I want to sample uniformly across e.g. [(kernel="linear"), (kernel="rbf"), (kernel="poly", degree=2), (kernel="poly", degree=3)]. Therefore, I was wondering if it is possible to conditionally introduce a hyperparameter for tuning, i.e. if kernel="poly" degree=np.linspace(2, 5, 4), else degree=0.

I haven't been able to find an example of this in the RandomizedSearchCV documentation, and so was wondering if anybody here had come across the same issue and would be able to help. Thanks!

from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import StratifiedKFold

clf = SVC()
params = {'kernel': ['linear', 'poly', 'rbf', 'sigmoid'],
          'degree': np.linspace(2, 5, 4),
          'C': np.logspace(-3, 5, 17),
          'gamma': np.logspace(-3, 5, 17)}

random_search = RandomizedSearchCV(
    estimator=clf, param_distributions=params, n_iter=200, n_jobs=-1,
    cv=StratifiedKFold(n_splits=5), iid=False
)

Upvotes: 3

Views: 2037

Answers (3)

Simon David
Simon David

Reputation: 1036

Since conditional paramaters are not supported in sklearn, the random parameter search will optimize a search space which includes redundant combinations of parameters. For example, there are already 289 combinations for kernal=linear for the given value ranges of C and gamma, adding degree will shoot this number up to 1156! In cases where model training takes a long time, you would want to omit these 867 useless combinations. The example below shows how you can provide your desired search space without any additional library.

n_c_values = len(np.logspace(-3, 5, 17))
n_gamma_values = len(np.logspace(-3, 5, 17))
n_degree_values = len(np.linspace(2, 5, 4))
print(n_c_values * n_gamma_values)
# 289
print(n_c_values * n_gamma_values * n_degree_values)
# 1156

sklearn allows us to pass multiple dictionaries of parameter grids to RandomizedSearchCV . In the example above, we thereby reduce the amount of parameter combinations from 4624 (4 * 1156) to 2023!

import numpy as np
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import StratifiedKFold

clf = SVC()

params = {'kernel': ['linear', 'rbf', 'sigmoid'],
          'C': np.logspace(-3, 5, 17),
          'gamma': np.logspace(-3, 5, 17)}

params_poly = {'kernel': ['poly'],
            'degree': np.linspace(2, 5, 4),
          'C': np.logspace(-3, 5, 17),
          'gamma': np.logspace(-3, 5, 17)}

param_grid = [params, params_poly]
# 2023 combinations

random_search = RandomizedSearchCV(
    estimator=clf, param_distributions=param_grid, n_iter=200, n_jobs=-1,
    cv=StratifiedKFold(n_splits=5)
)

Upvotes: 0

Amine Benatmane
Amine Benatmane

Reputation: 1261

Unfortunately, GridsearchCV and RandomizedSearchCV don't support conditional tuning of hyperparameters.

Hyperopt supports conditional tuning of hyperparameters, check this wiki for more details.

Example:

space4svm = {
    'C': hp.uniform('C', 0, 20),
    'kernel': hp.choice('kernel', [
            {'ktype': 'linear'},
            {'ktype': 'poly', 'degree': hp.lognormal('degree', 0, 1)},
            ]),
    'gamma': hp.uniform('gamma', 0, 20),
    'scale': hp.choice('scale', [0, 1]),
    'normalize': hp.choice('normalize', [0, 1])
}

Upvotes: 1

Celius Stingher
Celius Stingher

Reputation: 18367

I am not sure you can make conditional arguments for or within the gridsearch (it would feel like a useful feature). However, one solution to go around this, is to simply set all the hyperparameters for randomizesearchcv add make use of the errors_raise paramater, which will allow you to pass through the iterations that would normally fail and stop your process. Like this:

from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import StratifiedKFold

clf = SVC()
params = {'kernel': ['linear', 'poly', 'rbf', 'sigmoid'],
          'degree': np.linspace(2, 5, 4),
          'C': np.logspace(-3, 5, 17),
          'gamma': np.logspace(-3, 5, 17)}

random_search = RandomizedSearchCV(
    estimator=clf, param_distributions=params, n_iter=200, n_jobs=-1,
    cv=StratifiedKFold(n_splits=5), iid=False,errors_raise=0)

However

From sklearn's SVC documentation you shouldn't have any problems passing degree:

degree : int, optional (default=3) Degree of the polynomial kernel function (‘poly’). Ignored by all other kernels.

Upvotes: 1

Related Questions