scipy.optimize.minimze - SLSQP giving different results for bound and constrained problems

Question

Continuing from my title I compared the two methods in a specific problems. This doesn't happen for easy minimization problems so I will have to post my rather large problem at the bottom. Define

def objective(someArr):
    o = coeff.dot(someArr)-expected
    return o.dot(o)

And a constraint and some bounds which I thought were the same and mean everything needs to be none negative:

constr = {'type':'ineq','fun': lambda x: x.min()}
bounds = [(0,None) for _ in range(len(expected))]

Now minimize in two different ways (the guess is arbitrary):

guess = [1 for _ in range(len(expected))]
minimize(objective,guess,constraints=constr,method='slsqp')
minimize(objective,guess,bounds=bounds,method='slsqp')

Both succeed, but give different results! They are similar but the constrained version is seems to find a 'nicer solution' in the sense that it's continuous which I know is less likely (some of the initial guesses are left as is). It looks like the bounds version is giving a more reliable solution (including the discontinuity at the middle), any idea why? Why are the results different?

This doesn't happen for simple enough systems, So mine is below:

expected

array([  4.03735117e-02,   1.08637788e-02,   1.54964051e-02,
         3.78628505e-02,   2.37030325e-02,   6.86859818e-01,
         1.46476530e-01,   1.04440486e-03,   1.20711274e-01,
         1.60280666e-01,   2.33949054e-02,   8.46002796e-03,
         7.45780483e-04,   3.69392042e-02,   1.44665642e-03,
         1.44608308e-02,   1.27347734e-02,   4.23352961e-04,
         9.20542456e-02,   2.48495819e-03,   3.54655192e-01])

Coefficient matrix (21x21):

    [[  9.37707100e-20   6.68400440e-18   6.97912430e-16   8.98556840e-14
    1.07937057e-11   4.19849820e-10   2.73605280e-08   1.09170960e-06
    1.50449010e-05   7.96266900e-05   2.31555200e-04   4.57569000e-04
    5.90697100e-04   4.10655400e-04   1.63821000e-04   5.53010200e-05
    1.09951060e-05   3.84831400e-06   1.23909470e-06   2.31528300e-07
    5.02411300e-08]
 [  3.45748390e-15   1.16643780e-13   5.32307960e-12   1.01004120e-10
    4.05216220e-09   1.61456490e-07   4.23295710e-06   6.24295200e-05
    3.29901600e-04   6.66420600e-04   7.29823900e-04   5.27159300e-04
    2.27278100e-04   4.42585700e-05   4.29081900e-06   4.30265500e-07
    2.13397700e-08   4.01132800e-09   1.18220120e-09   1.31554450e-10
    6.78713900e-12]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   1.85846486e-13
    3.36231717e-10   1.85400562e-08   3.89468660e-07   5.55975320e-06
    4.73904880e-05   1.54447990e-04   1.50380210e-04   5.69735990e-05
    6.27187000e-06   9.82224200e-07   1.26135220e-07   1.16118950e-08
    1.28288250e-09]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    2.43215800e-14   5.96589295e-12   3.51565120e-09   1.40006520e-07
    3.61819670e-06   4.06318460e-05   1.32977260e-04   1.59613560e-04
    6.96288200e-05   3.19618580e-05   1.17693300e-05   2.59929440e-06
    6.08805400e-07]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    1.55452990e-11   6.12778810e-09   2.40761500e-07   3.84277600e-06
    2.97983700e-05   1.56114000e-04   5.30523000e-04   1.07510900e-03
    1.29982300e-03   8.48435000e-04   2.49909800e-04   4.34448700e-05
    2.72827100e-06   3.61865000e-07   5.30180000e-08   5.39343000e-09
    3.39819900e-10]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    9.80980000e-16   4.53215210e-13   5.93537000e-11   7.73680600e-09
    1.45490800e-07   2.06500700e-06   1.85582600e-05   1.01621100e-04
    3.65050000e-04   8.47504000e-04   9.16790000e-04   5.10335000e-04
    1.32094200e-04   5.01662000e-05   1.68653100e-05   3.42706000e-06
    7.64106000e-07]
 [  5.22393900e-16   9.57670700e-14   2.80224630e-11   6.17311800e-09
    3.63986700e-07   1.27199100e-05   1.61234500e-04   6.80855000e-04
    1.13993600e-03   1.09554200e-03   6.55816000e-04   2.29829600e-04
    4.92989000e-05   5.25123200e-06   1.41564430e-07   2.39038170e-09
    6.00416600e-12   6.19360000e-14   0.00000000e+00   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   4.75768800e-19   1.25757632e-16   3.85615412e-14
    1.64951977e-10   8.16844600e-09   1.44055600e-07   1.93038560e-06
    1.65309040e-05   7.25888800e-05   1.53391160e-04   1.85548480e-04
    1.41324560e-04   5.35520440e-05   9.08814400e-06   1.16508600e-06
    6.51461200e-08   9.63840800e-09   1.89874880e-09   2.80790120e-10
    1.77267480e-11]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   3.12433680e-18
    2.09937884e-15   6.15295200e-13   2.33827048e-10   9.42587360e-09
    2.18070840e-07   2.53094400e-06   1.39357400e-05   4.56638000e-05
    1.03468560e-04   1.37336160e-04   8.75893200e-05   3.60747400e-05
    7.94232400e-06   2.86150800e-06   9.40383600e-07   1.83250080e-07
    4.02287600e-08]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   2.78360280e-14
    1.13077826e-10   1.65225720e-08   4.49710800e-07   4.18136400e-06
    4.26663600e-06   1.14932400e-06   1.25458080e-07   2.73596400e-09
    9.87835200e-11]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   4.18941677e-09   6.46043040e-06   4.35858800e-05
    1.85997600e-05   3.43239200e-06   1.11978076e-07   5.19446488e-11
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   1.57210000e-12   3.60625836e-09
    3.30526520e-07   5.77426800e-06   1.33358320e-05   1.43552480e-06
    3.02920760e-08]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    2.92584400e-12   2.03562744e-08   3.00648720e-06   3.84799520e-06
    1.40841200e-07   4.38522680e-09   4.19531080e-11   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   8.04725600e-12   1.39600679e-07   7.36690800e-06
    2.59910000e-05   2.70401200e-05   4.35387960e-06   9.09141480e-09
    4.18099600e-12]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   7.54614320e-12
    4.96265360e-09   1.12553828e-06   1.37813920e-05   2.78272760e-06
    2.80655084e-09   3.01261760e-12   0.00000000e+00   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    7.84628000e-13   2.15015975e-08   1.03311776e-05   2.95554800e-05
    4.24798800e-06   3.50478760e-07   5.99805920e-09   2.26524600e-12
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   3.38659412e-10   1.68748108e-06   2.91162400e-05
    3.32621600e-05   1.37055520e-05   9.31179320e-07   7.59133436e-10
    1.19764000e-13]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    1.70595760e-13   8.46074200e-11   1.00050024e-08   5.02041840e-07
    9.72964000e-06   4.00182400e-05   2.37195920e-05   4.51005344e-07
    1.29503120e-11   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   8.50672000e-14   5.96485392e-09   9.45778520e-07
    1.02437720e-05   2.73561200e-05   1.05041560e-05   6.73162160e-08
    9.48935360e-11]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    1.55824024e-11   4.39452984e-08   3.48200400e-06   2.59034680e-06
    3.11364280e-08   3.22964640e-10   1.01458120e-12   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   3.09492760e-13   3.68745010e-10
    3.25169980e-08   1.59072640e-06   1.88484840e-05   5.89398200e-05
    5.35469200e-05   3.43275000e-05   1.51554420e-05   3.76945800e-06
    9.39956600e-07]]

Answer 1

It is not really surprising that different computations produce different results. Your constraints are mathematically equivalent to lower bounds, but in the SLSQP algorithm, they are treated differently. There is a pretty simple subroutine for enforcing upper/lower bounds, whereas the treatment of constraints is buried pretty deep in those 2150 lines of Fortran.

For an easy minimization problem this is less likely to matter, because the algorithm will likely find the best solution either way. But with difficult minimization problems you are never guaranteed the perfect result. The algorithm will run for a while, trying to improve the outcome, and will eventually decide to stop somewhere that looks like it might be the minimum. Where it will stop depends on all of the parameters you provide.

It's preferable to give numeric upper/lower bounds explicitly as bounds (not as a special case of an inequality constraint), because if a minimization routine accepts bounds separately from constraints, it probably has a more efficient way of dealing with them.