List of parameters in sklearn randomizedSearchCV like GridSearchCV?

Question

I have a problem where I'd like to test multiple models that don't all have the same named parameters. How would you use a list of parameters for a pipeline in RandomizedSearchCV like you can use in this example with GridSearchCV?

Example from:
https://scikit-learn.org/stable/auto_examples/compose/plot_compare_reduction.html

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
from sklearn.decomposition import PCA, NMF
from sklearn.feature_selection import SelectKBest, chi2

pipe = Pipeline([
    # the reduce_dim stage is populated by the param_grid
    ('reduce_dim', None),
    ('classify', LinearSVC())
])

N_FEATURES_OPTIONS = [2, 4, 8]
C_OPTIONS = [1, 10, 100, 1000]
param_grid = [
    {
        'reduce_dim': [PCA(iterated_power=7), NMF()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'classify__C': C_OPTIONS
    },
    {
        'reduce_dim': [SelectKBest(chi2)],
        'reduce_dim__k': N_FEATURES_OPTIONS,
        'classify__C': C_OPTIONS
    },
]

grid = GridSearchCV(pipe, cv=3, n_jobs=2, param_grid=param_grid)
digits = load_digits()
grid.fit(digits.data, digits.target)

Answer 1

This is an old issue that was resolved for a while now (not sure starting from which scikit-learn version).

You can now pass a list of dictionaries for RandomizedSearchCV in the param_distributions parameter. Your example code would become:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import RandomizedSearchCV
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
from sklearn.decomposition import PCA, NMF
from sklearn.feature_selection import SelectKBest, chi2

pipe = Pipeline([
    # the reduce_dim stage is populated by the param_grid
    ('reduce_dim', None),
    ('classify', LinearSVC())
])

N_FEATURES_OPTIONS = [2, 4, 8]
C_OPTIONS = [1, 10, 100, 1000]
param_grid = [
    {
        'reduce_dim': [PCA(iterated_power=7), NMF()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'classify__C': C_OPTIONS
    },
    {
        'reduce_dim': [SelectKBest(chi2)],
        'reduce_dim__k': N_FEATURES_OPTIONS,
        'classify__C': C_OPTIONS
    },
]

grid = RandomizedSearchCV(pipe, cv=3, n_jobs=2, param_distributions=param_grid)
digits = load_digits()
grid.fit(digits.data, digits.target)

I'm using sklearn version 0.23.1 .

Answer 2

Hyperopt supports Hyperparameter Tuning across multiple estimators, check this wiki for more details (2.2 A Search Space Example: scikit-learn section).

Check out this post if you want to use sklearn's GridSearch to do that. It suggests an implementation of EstimatorSelectionHelper estimator which can run different estimators, each with its own grid of parameters.

Answer 3

I have found a way around, that relies on duck-typing, and doesn't get too much in the way.

It relies on passing complete estimators as parameters to the pipeline. We first sample the kind of model, and then its parameters. For that we define two classes that can be sampled :

from sklearn.model_selection import ParameterSampler


class EstimatorSampler:
    """
    Class that holds a model and its parameters distribution.
    When sampled, the parameters are first sampled and set to the model, 
    which is returned.

    # Arguments
    ===========
    model : sklearn.base.BaseEstimator
    param_distributions : dict
        Input to ParameterSampler

    # Returns
    =========
    sampled : sklearn.base.BaseEstimator
    """
    def __init__(self, model, param_distributions):
        self.model = model
        self.param_distributions = param_distributions

    def rvs(self, random_state=None):
        sampled_params = next(iter(
            ParameterSampler(self.param_distributions, 
                             n_iter=1, 
                             random_state=random_state)))
        return self.model.set_params(**sampled_params)


class ListSampler:
    """
    List container that when sampled, returns one of its item, 
    with probabilities defined by `probs`.

    # Arguments
    ===========
    items : 1-D array-like
    probs : 1-D array-like of floats
        If not None, it should be the same length of `items`
        and sum to 1.

    # Returns
    =========
    sampled item
    """
    def __init__(self, items, probs=None):
        self.items = items
        self.probs = probs

    def rvs(self, random_state=None):
        item = np.random.choice(self.items, p=self.probs)
        if hasattr(item, 'rvs'):
            return item.rvs(random_state=random_state)
        return item

And the rest of the code is defined below.

    import numpy as np
    import matplotlib.pyplot as plt
    from sklearn.datasets import load_digits
    from sklearn.model_selection import RandomizedSearchCV
    from sklearn.pipeline import Pipeline
    from sklearn.svm import LinearSVC
    from sklearn.decomposition import PCA, NMF
    from sklearn.feature_selection import SelectKBest, chi2

    pipe = Pipeline([
        # the reduce_dim stage is populated by the param_grid
        ('reduce_dim', None),
        ('classify', None)
    ])

    N_FEATURES_OPTIONS = [2, 4, 8]
    dim_reducers = ListSampler([EstimatorSampler(est, {'n_components': N_FEATURES_OPTIONS})
                                for est in [PCA(iterated_power=7), NMF()]] + 
                               [EstimatorSampler(SelectKBest(chi2), {'k': N_FEATURES_OPTIONS})])

    C_OPTIONS = [1, 10, 100, 1000]
    classifiers = EstimatorSampler(LinearSVC(), {'C': C_OPTIONS})

    param_dist = {
        'reduce_dim': dim_reducers, 
        'classify': classifiers
    }

    grid = RandomizedSearchCV(pipe, cv=3, n_jobs=2, scoring='accuracy', param_distributions=param_dist)
    digits = load_digits()
    grid.fit(digits.data, digits.target)