Difference between Fitting Algorithms

Question

I am new to Mathematica, and I don´t quite understand, what the difference between the fitting algorithms Fit, FindFit, LinearModelFit, NonLinearModelFit are. Why is the syntax so different ({1,x,x^2} vs. a + b x + c x^2)? Which should I prefer for which use-case?
Why can I use a non linear Function like a + b x + c x^2 in an LinearModelFit? I am very confused, I hope you can help me.

Answer 1

Luca, I think that you may be misunderstanding the meaning of "linear" in the names of those functions, and in general in reference to linear models.

Just to make sure that we agree on the nomenclature here, in a fitting model you have parameters, and predictor variables. For instance, in a simple quadratic model such as y = a x^2 + b x + c we typically understand a, b, c as the parameters, and x as the predictor variable.

Linear models are those models in which the predicted value (e.g. y in our example) depends linearly on the parameters of the model. It doesn't matter whether the relationship between y and x is linear. For instance, the model above is linear. This following model is also linear: y = a Sin[x^2]. Model fitting is a search for the parameters, so linear parameter relationships lead to models that are easier to deal with computationally, and in some cases to closed-form solutions to the regression problem.

Non-linear models are those in which the predicted value depends on the parameters in a non-linear fashion. For instance: y = Sin[a] x is non linear because the predicted value depends non-linearly on the value of the parameter a (even though the predicted value depends linearly on x!).

In addition to the interesting reasons mentioned by the commenters to your question, which I think shed some light on the issue as well, the existence of multiple fitting functions also reflects this fundamental divide.

On the basis of my (limited) experience, here is how I view the most common fitting functions:

• Fit is the oldest of the bunch; it carries out linear model fitting by linearly combining a list of base functions you explicitly indicate. I don't tend to use it much, since the majority of models I fit to are non-linear.
• FindFit is newer; it is capable of both linear and non-linear fitting. You feed it the explicit form of the fitting model directly. I find this very intuitive to use.
• NonlinearModelFit accomplishes the same results as FindFit and it shares the same syntax. In addition to the fitted model, it also gives you access to a wealth of fitting descriptors: goodness of fit estimates, prediction intervals, residuals, just to name a few. You could think of it as a FindFit "on steroids". It would be really worth your time to take a look at the many options available for this function ("Details and Options" section of its documentation).
• FindFit is also faster than NonlinearModelFit: the former is implemented in fast numerical code within the kernel; the latter is written entirely in top-level code, i.e. using Mathematica expressions, and it internally calls FindFit to do the heavy lifting anyway (@OleksandrR. talked about this point more authoritatively in his answer regarding compiled versions of the fitting routines).