CODEWITHSUNDEEP
pythonpandasscipy
Stefano Potter
Stefano Potter

Reputation: 3577

Scipy Optimized Curve - warning the covariates could not be estimated

I have a dataset like this:

    elevation  Tree_cover  dNBR  below_ground_carbon_combusted     DOB_lst
0  266.444444   39.555556  0.47                    1110.603930  221.879608
1  683.222222    7.555556  0.48                    2901.511339  236.847916
2  270.777778   44.111111  0.54                    5822.440530  222.909241
3  473.444444   76.333333  0.73                    4643.891015  183.100876
4  470.666667   79.888889  0.64                    5919.393383  183.100876
5  486.333333   76.444444  0.68                    5120.039877  183.100876
6  453.000000   79.222222  0.63                    7090.889123  183.100876
7  412.555556   73.888889  0.48                    2636.000936  183.100876
8  628.000000   45.444444  0.60                   10075.698853  195.514770
9  631.555556    3.000000  0.45                    6144.187670  178.886703

And I want to fit a curve to the data with scipy's optimized curve like so:

from scipy.optimize import curve_fit
from math import e
import pandas as pd

def func(data, c1, c2, c3, c4, c5, c6, c7):

    var1 = (-(df['elevation'] - c6)**2)
    var2 = 2 * c7
    var3 = var1 / var2
    return (c1 + c2) * (e**c3 * df['dNBR']) * (e**c4 * df['Tree_cover']) * (e**c5 * df['DOB_lst']) * e**var3


popt, pcov = curve_fit(func, df.drop(['below_ground_carbon_combusted'], axis = 1), df['below_ground_carbon_combusted'])

but I can an error saying the curve can't be optimized and I can't figure out why since with other functions (which are simpler) this general workflow works fine.

This is the full error:

OptimizeWarning: Covariance of the parameters could not be estimated
  category=OptimizeWarning)

Upvotes: 2

Views: 149

Answers (1)

R.yan
R.yan

Reputation: 2372

Why? Basically, it is caused by the return values, since your function return a zero array over every iteration.

And the reason why result is zero is because e**var3. Since var3 is highly negative, so e**var3 would appoach to zero

How to fix it? Normalize your data before do the optimization.

from sklearn import preprocessing

x = df.values #returns a numpy array
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x)
df = pd.DataFrame(x_scaled,columns=['elevation','Tree_cover','dNBR','below_ground_carbon_combusted','DOB_lst'])
print df
below = df.copy()

P.S MinMaxScaler is just one way to do normalization. You can choose other way.

Output

print popt
print pcov
[  46.64430112   46.64430112   -9.82987816 -161.61131905  128.11954276
    0.7539214     1.03938635]
[[ -3.08229583e+55   1.26754744e+56  -2.40802563e+53   3.74896120e+53
    1.29697774e+54  -2.80789589e+54   1.60677922e+54]
 [  8.98526683e+55   8.98526683e+55   7.01973971e+53  -1.04507188e+54
   -1.40394794e+54   2.80789589e+54   8.42368766e+54]
 [  0.00000000e+00  -0.00000000e+00   2.19366866e+52   1.14667678e+51
    0.00000000e+00   0.00000000e+00   8.77467464e+52]
 [  1.40394794e+54  -1.40394794e+54   1.09683433e+52  -2.25100004e+52
    0.00000000e+00   0.00000000e+00  -4.38733732e+52]
 [ -0.00000000e+00   0.00000000e+00  -2.19366866e+52   2.07900045e+52
   -2.19366866e+52   0.00000000e+00  -8.77467464e+52]
 [ -2.80789589e+54   2.80789589e+54  -2.19366866e+52  -1.14667678e+51
    0.00000000e+00   0.00000000e+00  -8.77467464e+52]
 [  0.00000000e+00   1.12315835e+55   0.00000000e+00  -3.26492254e+52
   -8.77467464e+52   0.00000000e+00   0.00000000e+00]]

Upvotes: 3

Related Questions