CODEWITHSUNDEEP
pythonclassstoragepickledeap
sdgaw erzswer
sdgaw erzswer

Reputation: 2382

Storing Deap models as pickle objects

I've been recently trying to first create a class which conducts some form of evolution based on the Deap library (https://deap.readthedocs.io/en/master/), and once trained, it is stored as a pickle object. The minimal working example of the issue I'm facing is the following one:

## minimal test
import random
from deap import base, creator, tools
import dill as pickle

def store_model(model_class, path, verbose = True):
    
    f = open(path, "wb")
    pickle.dump(model_class, f)
    f.close()
    
    if verbose:
        print("Stored the model info!")
    

def load_model(path, verbose = True):

    
    f = open(path, "rb")
    unpickled_model = pickle.load(f)
    f.close()

    return unpickled_model

class TestClass:

    def __init__(self):
        self.creator = creator
        self.toolbox = base.Toolbox()
        self.total_params = 5
        self.base = base


    def mutReg(self, individual, p=1):
        """
        Custom mutation operator used for regularization optimization.

        :param individual: individual (vector of floats)
        :return individual: An individual solution.
        """

        individual[0] += random.random() * self.reg_constant
        return individual,
        
    def somefun(self):
        
        self.toolbox.register("attr_float", random.uniform, 0.00001, 0.999999)
        self.creator.create("FitnessMulti", self.base.Fitness, weights=(1.0, ))
        self.creator.create("Individual", list, fitness=self.creator.FitnessMulti)

        self.toolbox.register("attr_float", random.uniform, 0.00001, 0.999999)
        self.toolbox.register("individual",
                         tools.initRepeat,
                         self.creator.Individual,
                         self.toolbox.attr_float,
                         n=self.total_params)
        self.toolbox.register("population",
                         tools.initRepeat,
                         list,
                         self.toolbox.individual,
                              n=100)
        self.toolbox.register("mate", tools.cxUniform, indpb=0.5)
        self.toolbox.register("mutate",
                         tools.mutGaussian,
                         mu=0,
                         sigma=0.2,
                         indpb=0.2)
        self.toolbox.register("mutReg", self.mutReg)
        self.toolbox.register("select", tools.selTournament)




clx = TestClass()
clx.somefun() # initialize the evolution classes

store_model(clx, "test.pickle") # Store as a pickled class

model = load_model("test.pickle") # Load as a class

print(model)

Here, if I just run this as python minimal_test_load.py , it works fine, outputting the stored class:

Stored the model info!
<__main__.TestClass object at 0x7f640308d190>

However, if I try to call load_model(pickled_object_path) from a different python file, it throws:

AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/home/someuser/miniconda3/envs/someenv/lib/python3.9/site-packages/deap-1.3.1-py3.9-linux-x86_64.egg/deap/creator.py'>

The problem appears to be the additional creation of new classes by the deap library that are not recognized by the pickle if the environment changes. Are there any known solutions to this type of issue? What I am trying to achieve is to just load a pickled object and be able to run some (of its) functions from any new python script (or notebook for example).

Thanks!

Upvotes: 0

Views: 368

Answers (1)

sdgaw erzswer
sdgaw erzswer

Reputation: 2382

I was able to circumvent this problem (with working results as follows):

when importing the creator, I globally created the

creator.create("FitnessMulti", self.base.Fitness, weights=(1.0, ))
creator.create("Individual", list, fitness=self.creator.FitnessMulti)

attributes. By doing the same when loading the pickled object somewhere else, these class attributes were already defined when loading the object (see the load_model in the original question), which works flawlessly. If there is a more elegant solution than creating a global variable, I'd still like to know!

Upvotes: 1

Related Questions