CODEWITHSUNDEEP
pythonscipy
bernie
bernie

Reputation: 586

scipy.optimize.minimize, needs more than one value to unpack with analytical jacobian

I'd like to minimize a 2D scalar function along a single direction, i.e. a line search. e.g. I'd like to do a 2D scalar version of:

scipy.optimize.minimize(lambda x: x*x, 2, jac=lambda x: 2*x)

But the following doesn't work for me. (Using a test function f(x,y) = (x-4)^2 + (y-4)^2. )

def my_func((x,y)):
    return (x-4)**2+(y-4)**2

def grad_my_func((x,y)):
    return (2*(x-4), 2*(y-4))

scipy.optimize.minimize works if I do the following:

mygrad = np.array(grad_my_func((-4,-4)))
# mygrad -> [-16,-16]
scipy.optimize.minimize(lambda x: my_func(np.array([-4,-4]) + x*mygrad), 0)

result is such that [-4,-4] + x*mygrad produces the minimum at (4,4):

   status: 0
  success: True
     njev: 4
     nfev: 12
 hess_inv: array([[ 0.00097656]])
      fun: 2.842170943040401e-14
        x: array([-0.50000001])
  message: 'Optimization terminated successfully.'
      jac: array([ 0.])
      nit: 2

But if I try to put in a function for the directional derivative, it doesn't work, and I don't understand why. I'm still technically giving 1D functions to the minimizer right?

scipy.optimize.minimize(
    lambda x: my_func(np.array([-4,-4]) + x*mygrad), 0, 
    jac=lambda x: np.dot(
        grad_my_func(np.array([-4,-4]) + x*mygrad),
        mygrad/scipy.linalg.norm(mygrad)) )

Long traceback. What i can't decipher about the traceback is that it points at my jac function, but the final complaint is about my_func, which jac doesn't call (uses grad_my_func). 

ValueError                                Traceback (most recent call last)
<ipython-input-66-1e528ef250f5> in <module>()
      1 scipy.optimize.minimize(
      2     lambda x: my_func(np.array([-4,-4]) + x*mygrad), 0,
----> 3     jac=lambda x: np.dot(
      4         grad_my_func(np.array([-4,-4]) + x*mygrad),
      5         mygrad/scipy.linalg.norm(mygrad)) )

/usr/local/lib/python2.7/site-packages/scipy/optimize/_minimize.pyc in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
    436         return _minimize_cg(fun, x0, args, jac, callback, **options)
    437     elif meth == 'bfgs':
--> 438         return _minimize_bfgs(fun, x0, args, jac, callback, **options)
    439     elif meth == 'newton-cg':
    440         return _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,

/usr/local/lib/python2.7/site-packages/scipy/optimize/optimize.pyc in _minimize_bfgs(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, **unknown_options)
    859             alpha_k, fc, gc, old_fval, old_old_fval, gfkp1 = \
    860                      _line_search_wolfe12(f, myfprime, xk, pk, gfk,
--> 861                                           old_fval, old_old_fval)
    862         except _LineSearchError:
    863             # Line search failed to find a better solution.

/usr/local/lib/python2.7/site-packages/scipy/optimize/optimize.pyc in _line_search_wolfe12(f, fprime, xk, pk, gfk, old_fval, old_old_fval, **kwargs)
    693     ret = line_search_wolfe1(f, fprime, xk, pk, gfk,
    694                              old_fval, old_old_fval,
--> 695                              **kwargs)
    696
    697     if ret[0] is None:

/usr/local/lib/python2.7/site-packages/scipy/optimize/linesearch.pyc in line_search_wolfe1(f, fprime, xk, pk, gfk, old_fval, old_old_fval, args, c1, c2, amax, amin, xtol)
     99     stp, fval, old_fval = scalar_search_wolfe1(
    100             phi, derphi, old_fval, old_old_fval, derphi0,
--> 101             c1=c1, c2=c2, amax=amax, amin=amin, xtol=xtol)
    102
    103     return stp, fc[0], gc[0], fval, old_fval, gval[0]

/usr/local/lib/python2.7/site-packages/scipy/optimize/linesearch.pyc in scalar_search_wolfe1(phi, derphi, phi0, old_phi0, derphi0, c1, c2, amax, amin, xtol)
    172         if task[:2] == b'FG':
    173             alpha1 = stp
--> 174             phi1 = phi(stp)
    175             derphi1 = derphi(stp)
    176         else:

/usr/local/lib/python2.7/site-packages/scipy/optimize/linesearch.pyc in phi(s)
     85     def phi(s):
     86         fc[0] += 1
---> 87         return f(xk + s*pk, *args)
     88
     89     def derphi(s):

/usr/local/lib/python2.7/site-packages/scipy/optimize/optimize.pyc in function_wrapper(*wrapper_args)
    283     def function_wrapper(*wrapper_args):
    284         ncalls[0] += 1
--> 285         return function(*(wrapper_args + args))
    286
    287     return ncalls, function_wrapper

<ipython-input-66-1e528ef250f5> in <lambda>(x)
      1 scipy.optimize.minimize(
----> 2     lambda x: my_func(np.array([-4,-4]) + x*mygrad), 0,
      3     jac=lambda x: np.dot(
      4         grad_my_func(np.array([-4,-4]) + x*mygrad),
      5         mygrad/scipy.linalg.norm(mygrad)) )

<ipython-input-34-76d10d8be6fa> in my_func(***failed resolving arguments***)
----> 1 def my_func((x,y)):
      2    return (x-4)**2+(y-4)**2

ValueError: need more than 1 value to unpack

Upvotes: 0

Views: 663

Answers (2)

bernie
bernie

Reputation: 586

Turns out that ValueError: need more than 1 value to unpack was truly the error. So the scipy routine was returning [[j]] as the value of x that minimizes f(x + x*grad), then I would multiply using x*grad and get [[ x*grad[0], x*grad[1] ]]. But my function was expecting a tuple-like structure ([ x*grad[0], x*grad[1] ]), not a list in a tuple.

Revising the code to the following:

scipy.optimize.minimize(
    lambda x: my_func(np.array([-4,-4]) - x[0]*mygrad), [0.], 
    jac=lambda x: np.dot(
        grad_my_func(np.array([-4,-4]) - x[0]*mygrad),
        mygrad/scipy.linalg.norm(mygrad)) )

i.e. [0.] instead of 0 as initial guess,x[0]*mygrad instead of x*mygrad, resolved the issue.

I can also add that I think my analytical version of the directional derivative is correct, but that supplying the derivative in function form seems to lead to more numerical problems than just having the derivative be estimated.

Upvotes: 0

hpaulj
hpaulj

Reputation: 231345

When I give your lambdas names, and use those in the command, the error stack is:

In [184]: optimize.minimize(bar,0,jac=foo1)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-184-ef7e300bda33> in <module>()
----> 1 optimize.minimize(bar,0,jac=foo1)

/usr/lib/python3/dist-packages/scipy/optimize/_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
    439         return _minimize_cg(fun, x0, args, jac, callback, **options)
    440     elif meth == 'bfgs':
--> 441         return _minimize_bfgs(fun, x0, args, jac, callback, **options)
    442     elif meth == 'newton-cg':
    443         return _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,
 etc

In your error message the ---> isn't pointing specifically as the jac parameter. It is just pointing to the middle of your multiline expression:

      1 scipy.optimize.minimize(
      2     lambda x: my_func(np.array([-4,-4]) + x*mygrad), 0,
----> 3     jac=lambda x: np.dot(
      4         grad_my_func(np.array([-4,-4]) + x*mygrad),
      5         mygrad/scipy.linalg.norm(mygrad)) )

The error could still have something to do with what jac returns. If it returns the wrong number of terms it could change the shape of x0, and give problems the next time my_func is called. This is suggested by the fact that the call runs if the jac is not defined

I'd try modeling the analytical jac after the numeric estimate:

      fun: 2.842170943040401e-14
 hess_inv: array([[ 0.00097656]])
      jac: array([ 0.])
  message: 'Optimization terminated successfully.'

In other words, it must return a single element.

Upvotes: 1

Related Questions