CODEWITHSUNDEEP
openmdao
Charlie Tran
Charlie Tran

Reputation: 11

How to choose the right optimizer?

I encountered a wrong optimum when minimizing Rosenbrock's function : 100(y-x**2)**2 + (1-x)**2. The correct result is x=y=1.0, f = 0. But when using OpenMDAO with COBYLA optimizer, the result was x=0.5660643, y=0.3179899, f=0.18889496. I switched to Powell optimizer and the process converged to the right result. So, how to choose the consistent optimizer for an optimization problem?

Upvotes: 1

Views: 506

Answers (2)

user2375049
user2375049

Reputation: 350

Sorry, I was not allowed to add a comment so this comes as an answer though it is a follow up question;

maybe it is important to ask/mention if the normalization of parameters (either objective(s) or design variables) changes the convergence depending on the optimizer. In other words does it matter more for a gradient based optimizer that the variables are normalized than the gradient free ones? What is the effect of scaling and normalizing for a single objective case (assuming for a multiobjective case it is clear why one should normalize) ?

Upvotes: 0

Justin Gray
Justin Gray

Reputation: 5710

there is, unfortunately, no good answer to this question. In fact, Wolpert and Macready proved that you can't answer this question definitively with their no free lunch theorem that says no optimizer can perform the best across all problems.

I will give you some broad suggestions: You need to understand the nature of the problem you're solving.

  1. Is it bumpy or convex(ish)?
  2. Is it continuous or discrete
  3. Do you have constraints
  4. How many design variables do you have? How many constraints?
  5. Can you get analytic derivatives or do you need to approximate them?
  6. If you need to approximate them can you use finite-difference or complex-step (complex step is better!)

Given all of that, you can start to narrow down your choices. Most of the time engineering problems are convex-ish (super technical term!). By that I mean that, while you can't prove there is more than one optimum, practically speaking the optimum you find tends to be insensitive to your starting point. If this is the case, you are better off using a gradient based optimizer rather than a gradient free one. COBYLA is a gradient free method. SLSQP and CONMIN are both gradient based.

Simply switching to a gradient based method won't necessarily solve your accuracy problems. These methods need accurate derivatives. If you don't define analytic derivatives, and instead use FD, then the approximations might not be enough to converge to the correct answer. They might be ok though, if you play with step sizes a bit. In general, FD will cause at best slower convergence and at worst prevent you from getting the answer.

If you want the most reliable optimizations, then you should use gradient based optimization with analytic derivatives or complex-step approximated derivatives (much more accurate than FD). For large numbers of design variables, or for problems that have expensive nonlinear solver loops, analytic derivatives will be much much faster.

If you know your problem has multiple local minima, then you need to use either a multi-start gradient based approach or a gradient free approach.

Upvotes: 4

Related Questions