Minimize error with an optimization approach similarly to solver

Question

I'm tryning to change all excel reports at my work to python scripts and I have a problem when trying to optimize a function to minimize the error.

What i'm doing here is calculate a weibull analysis to predict how much periods is going to last a product, the thing is that to make this prediction I use an alpha and beta. This params in excell are generated through "solver min nonlinear" by minimizing the total_error. This total error is the sum of abs error resulting from the substraction between the real observed values in periods and the predicted values

Observed Predicted
22.0    19
28.0    26
29.0    22
29.0    24
29.0    28
30.0    31
31.0    21

ABS_ERROR
0.03
0.02
0.07
0.05
0.01
0.01
0.1

TOTAL_ERROR:0.29

def gen_perm(tsd):
    perm = tsd[0].rolling(min_periods=1, window=len(tsd[0])).sum()
    perm = perm.reindex(index=perm.index[::-1])
    perm = perm.reset_index()
    perm.GENERACION = [x for x in range(1,len(perm)+1)]
    
    sums = []
    for i in tsd.columns:
        if i != 0:
            x = tsd[i].sum()
            sums.append(x)
        else:
            None
            
    perm['POL_COB_EFECT'] = sums
    perm.rename(columns={'GENERACION':'PERIODOS',0:'PRODUCTS'},inplace=True)
    perm['PERM_OBS'] = (perm.PRODUCTS_COB_EFECT / perm.PRODUCTS * 100).astype(int)
    
    return perm

def perm_proy(alfa, beta, row):
    proy_l = []
    for i in range(len(row['PERIODOS'])):
        p_proy = (exp(-alfa*pow(i, beta)))*(row['PERM_OBS'].iloc[0] / 100)
        proy_l.append(int(p_proy * 100))
        
    return proy_l

test['ABS_ERROR'] =abs((test['PERM_OBS'] - test['PERM_PROY'])/100)

alfa = 0.17177273010
beta = 0.78239709633

def abs_error(alfa, beta, df):   # ---> tHIS IS THE ONE THAT i'M TRYING TO OPTIMIZE BY CHANGING THE VALUES FOR ALPFA AND BETA
    
    df['PERM_PROY'] = perm_proy(alfa, beta, df)
    df['ABS_ERROR'] = abs((df['PERM_OBS'] - df['PERM_PROY'])/100)
    
    error = df['ABS_ERROR'].sum()
    
    return error

At the end what I'm looking is to change the values for alfa and beta in order to minimize the error.

EDIT

'Till now I follow and example of scipy.optimize.minimize and this is my code, but at the end instead og giving me another values for alfa and beta it keeps giving me the initial_guess:

import scipy.optimize as optimize



def abs_error(params):
    alfa, beta = params
    df['PERM_PROY'] = perm_proy(alfa, beta, df)
    df['ABS_ERROR'] = abs((df['PERM_OBS'] - df['PERM_PROY'])/100)
    
    error = df['ABS_ERROR'].sum()
    
    return error
 



initial_guess = [0.17177273010,  0.78239709633]
result = optimize.minimize(abs_error, initial_guess)
if result.success:
    fitted_params = result.x
    print(fitted_params)
else:
    raise ValueError(result.message)

Answer 1

People will be able to help much better if you give a full example that can be run, but picking two values isn't a very difficult problem so common approaches should work.

Scipy

If you don't mind using dependencies, scipy comes with a bunch of optimizers under scipy.optimize. You can find the API reference here.

If you end up going the scipy route, you will basically need to write a function that takes a parameters array and returns an error value to minimize. Most of the code you have written should be applicable.

Writing it yourself

If you don't want to introduce a dependency, or if you want to learn how to do it yourself, it is not too complex to write a basic optimizer.

The simplest approach that will still give decent results is probably a Hill Climbing algorithm. You can find a short description of it here and here.

I'll just summarize it here.

1. Start with an initial solution. (Can be any value, random is a good idea)
2. Make a small mutation (Add or subtract small random values from the parameters)
3. If the error is less than or equal to the best solution, keep it. Otherwise, revert back to the best solution.
4. Repeat until X generations have passed, or the error is sufficiently small.