Methods for NonlinearModelFit

Question

I am using NonlinearModelFit for a thesis project.

I get quite different results if I change the Method to LevenbergMarquardt or QuasiNewton or ConjugateGradient and so on. My nonlinear model includes polynomials up to second order and a sigmoid function such as ArcTan or Tanh. I would like to know, which conditions Mathematica uses to choose the best algorithm if I set the Method as Automatic. It works sometimes, but most of the time it doesn't and I do not have the knowledge to say which method should be the best in my case. Just by trying out, I found Gradient and sometimes ConjugateGradient to work the best for my purpose.

I also cannot find anywhere on the Internet which exact implementation of these methods is used inside Mathematica and how does it change the results. Here maybe an interesting link: Optimization Algorithms

Thanks!

Answer 1

Unfortunately the Method option is not documented in detail on the NonlinearModelFit documentation page. To summarize what we know so far (comments, documentation, etc.):

NonlinearModelFit can either use numerical local optimization, or numerical global optimization.

• Local optimization is the same as used by FindMinimum and related functions. The possible method options are documented in detail here. NonlinearModelFit will take these options directly. Example:

  NonlinearModelFit[..., Method -> {"Newton", "StepControl" -> "TrustRegion"}]
  

• Global optimization is the same as used by NMinimize and related functions. The available method options are detailed here. The syntax to use is

  NonlinearModelFit[..., Method -> {NMinimize, Method -> ...}]
  

Further documentation on the constrained optimization is here.