SVM Optimisation using Niapy

Question

Dear good souls of this Platform,

Please I am so much disturbed that my head is like going to explode because I cannot understand how to use any of Niapy algorithm (like PSO) to optimise SVM parameters. I spend two days trying to figure out how to do it but I couldn't. The example on the Niapy Github site is not that very clear for me to understand.

Below is the code from Niapy

import sys
sys.path.append('../')

from NiaPy.task import StoppingTask, OptimizationType
from NiaPy.benchmarks import Benchmark
from NiaPy.algorithms.basic import GreyWolfOptimizer

##################################################################################
### our custom benchmark class   
class MyBenchmark(Benchmark):
    def __init__(self):
        Benchmark.__init__(self, -10, 10)

    def function(self):
        def evaluate(D, sol):
            val = 0.0
            for i in range(D): val += sol[i] ** 2
            return val
        return evaluate
####################################################################################

### we will run 10 repetitions of Grey Wolf Optimizer against our custom MyBenchmark benchmark function

for i in range(10):
    task = StoppingTask(D=20, nGEN=100, optType=OptimizationType.MINIMIZATION, benchmark=MyBenchmark())

    # parameter is population size
    algo = GreyWolfOptimizer(NP=20)

    # running algorithm returns best found minimum
    best = algo.run(task)

    # printing best minimum
    print(best[-1])

The above code works fine. I want to replace MyBenchmark class (demarcated with ###) with the code below:

def SVR_PSO(params):
    global PSO_model    

    PSO_model = SVR(C=params[0][0], epsilon = params[0][1], gamma = params[0][2])
    PSO_model.fit(X_train,np.ravel(y_train))
    result = PSO_model.predict(X_val)
    MAPE_result = call_MAPE(y_val, result)   
    # print('New PSO: C = {c}, epsilon={e}, gamma={g}  MAPE={m}'.format(c=params[0][0], e=params[0][1], g=params[0][2], m=MAPE_result)) #, g=params[2],  m=resultCV))
    #print(params)
    return MAPE_result

Thank you all.

Please, please please someone please help me.

Answer 1

The API has changed quite a bit since this question was asked. I apologize for the unclear documentation, we're working on making it better.

First thing you have to do, is implement a custom Problem class (previously Benchmark). The optimization problem will have 3 dimensions, because there's 3 parameters, the lower and upper bounds will be the lower and upper bounds of those parameters, and it has to store the dataset:

from niapy.problems import Problem
from niapy.task import Task
from niapy.algorithms.basic import ParticleSwarmOptimization

from sklearn.svm import SVR

def preprocess_data(dataset):
    """Split, scale, whatever..."""
    pass

class SVMHyperparameterOptimization(Problem):
    def __init__(lower, upper, dataset):
        super().__init__(dimension=3, lower=lower, upper=upper)
        self.X_tr, self.y_tr, self.X_val, self.y_val = preprocess_data(dataset)
        # or do the preprocessing here...
    
    # This is the method containing the objective function.
    def _evaluate(x):
        model = SVR(C=x[0], epsilon=x[1], gamma=x[2])
        model.fit(self.X_tr, self.y_tr)
        result = model.predict(self.X_val)
        mape = call_MAPE(self.y_val, result)
        return mape

Then you run the algorithm on the problem like so:

# Load dataset...

# set lower and upper bounds
# lower[0] => min C, lower[1] => min gamma, lower[2] => min epsilon
# upper[0] => max C, upper[1] => max gamma, upper[2] => max epsilon
# lower and upper can be tuples, lists, numpy arrays or scalars. If they're scalars, each parameter will have the same bound.
lower = (0.01, 0.0001, 0.0001)
upper = (32000.0, 32.0, 1.0)

problem = SVMHyperparameterOptimization(lower, upper, dataset)

"""
The task class is basically a wrapper for Problem, it counts iterations and
function evaluations and evaluates stopping conditions. Here I set the max number
of iterations to 500, you can also set max_evals, which is the max number of 
function evaluations, and a cutoff_value, so the algorithm will stop running if it
reaches that value.
"""
task = Task(problem=problem, max_iters=500)

# Initialize the PSO algorithm
pso = ParticleSwarmOptimization(population_size=10, c1=2.0, c2=2.0)

# Run the algorithm on task
best_params, best_mape = pso.run(task)
C, gamma, epsilon = best_params
print('Best parameters: C={}; gamma={}; epsilon={}'.format(C, gamma, epsilon))
print('Best score:', best_mape)