Hyperparameter Optimization in sklearn and n_jobs > 1: Pickling

Question

I'm in a "pickle". Here's the structure of my code:

• A base class that acts like an abstract class
• A subclass that can be instantiated
  • A method that sets up the parameters and calls RandomizedSearchCV or GridSearchCV with n_jobs=-1.
    • A local function, create_model, that creates the neural network model (see this tutorial) to be called by KerasClassifier or KerasRegressor

I get an error saying local object can't be pickled. If I change n_jobs=1, then no problems. So I suspect the issue is with the local function and parallel processing. Is there a fix to this? After googling a bit, it seems that the serializer dill could work here (I even found a package called multiprocessing_on_dill). But I'm currently relying on sklearn's packages.

Answer 1

I can confirm the same problem when running sklearn's grid search on a kerasClassifier model with parallelization (n_jobs>1) on Windows in a jupyter notebook/ipython (no problem on Unix).

I solved the issue by putting the create_model function that causes the pickle problem into a module and importing the module instead of defining the function within the environment.

To create a simple module for Python,

• create a text file in the same folder you're running the main code from and save it as my_module.py
• put the definition of the create_model function into the file
• instead of defining create_model in your code, import the module with import my_module and call your function from the module with my_module.create_model()

Answer 2

I found a "solution" to my problem. I was really puzzled why the examples here work with n_jobs=-1, but my code doesn't. It seems the issue is with the local function create_model that resides in a method of the subclass. If I make the local function a method of the subclass, I'm able to set n_jobs > 1.

So to recap, here's the structure of my code:

• A base class that acts like an abstract class
• A subclass that can be instantiated
  • A method that sets up the parameters and calls RandomizedSearchCV or GridSearchCV with n_jobs=-1.
  • A method, create_model, that creates the neural network model to be called by KerasClassifier or KerasRegressor

General idea of the code:

from abc import ABCMeta
import numpy as np
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

class MLAlgorithms(metaclass=ABCMeta):

    def __init__(self, X_train, y_train, X_test, y_test=None):
        """
        Constructor with train and test data.
        :param X_train: Train descriptor data
        :param y_train: Train observed data
        :param X_test: Test descriptor data
        :param y_test: Test observed data
        """
        ...

    @abstractmethod
    def setmlalg(self, mlalg):
        """
        Sets a machine learning algorithm.
        :param mlalg: Dictionary of the machine learning algorithm.
        """
        pass

    @abstractmethod
    def fitmlalg(self, mlalg, rid=None):
        """
        Fits a machine learning algorithm.
        :param mlalg: Machine learning algorithm
        """
        pass


class MLClassification(MLAlgorithms):
    """
    Main class for classification machine learning algorithms.
    """

    def setmlalg(self, mlalg):
        """
        Sets a classification machine learning algorithm.
        :param mlalg: Dictionary of the classification machine learning algorithm.
        """
        ...

    def fitmlalg(self, mlalg):
        """
        Fits a classification machine learning algorithm.
        :param mlalg: Classification machine learning algorithm
        """
        ...

    # Function to create model, required for KerasClassifier
    def create_model(self, n_layers=1, units=10, input_dim=10, output_dim=1,
                     optimizer="rmsprop", loss="binary_crossentropy",
                     kernel_initializer="glorot_uniform", activation="sigmoid",
                     kernel_regularizer="l2", kernel_regularizer_weight=0.01,
                     lr=0.01, momentum=0.0, decay=0.0, nesterov=False, rho=0.9, epsilon=1E-8,
                     beta_1=0.9, beta_2=0.999, schedule_decay=0.004):
        from keras.models import Sequential
        from keras.layers import Dense
        from keras import regularizers, optimizers

        # Create model
        if kernel_regularizer.lower() == "l1":
            kernel_regularizer = regularizers.l1(l=kernel_regularizer_weight)
        elif kernel_regularizer.lower() == "l2":
            kernel_regularizer = regularizers.l2(l=kernel_regularizer_weight)
        elif kernel_regularizer.lower() == "l1_l2":
            kernel_regularizer = regularizers.l1_l2(l1=kernel_regularizer_weight, l2=kernel_regularizer_weight)
        else:
            print("Warning: Kernel regularizer {0} not supported. Using default 'l2' regularizer.".format(
                kernel_regularizer))
            kernel_regularizer = regularizers.l2(l=kernel_regularizer_weight)

        if optimizer.lower() == "sgd":
            optimizer = optimizers.sgd(lr=lr, momentum=momentum, decay=decay, nesterov=nesterov)
        elif optimizer.lower() == "rmsprop":
            optimizer = optimizers.rmsprop(lr=lr, rho=rho, epsilon=epsilon, decay=decay)
        elif optimizer.lower() == "adagrad":
            optimizer = optimizers.adagrad(lr=lr, epsilon=epsilon, decay=decay)
        elif optimizer.lower() == "adadelta":
            optimizer = optimizers.adadelta(lr=lr, rho=rho, epsilon=epsilon, decay=decay)
        elif optimizer.lower() == "adam":
            optimizer = optimizers.adam(lr=lr, beta_1=beta_1, beta_2=beta_2, epsilon=epsilon, decay=decay)
        elif optimizer.lower() == "adamax":
            optimizer = optimizers.adamax(lr=lr, beta_1=beta_1, beta_2=beta_2, epsilon=epsilon, decay=decay)
        elif optimizer.lower() == "nadam":
            optimizer = optimizers.nadam(lr=lr, beta_1=beta_1, beta_2=beta_2, epsilon=epsilon,
                                         schedule_decay=schedule_decay)
        else:
            print("Warning: Optimizer {0} not supported. Using default 'sgd' optimizer.".format(optimizer))
            optimizer = "sgd"

        model = Sequential()
        model.add(
            Dense(units=units, input_dim=input_dim,
                  kernel_initializer=kernel_initializer, activation=activation,
                  kernel_regularizer=kernel_regularizer))
        for layer_count in range(n_layers - 1):
            model.add(
                Dense(units=units, kernel_initializer=kernel_initializer, activation=activation,
                      kernel_regularizer=kernel_regularizer))
        model.add(Dense(units=output_dim,
                        kernel_initializer=kernel_initializer, activation=activation,
                        kernel_regularizer=kernel_regularizer))

        # Compile model
        model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])
        return model


class MLRegression(MLAlgorithms):
    """
    Main class for regression machine learning algorithms.
    """
    ...