Using NonlinearModelFit to fit data with errors

Question

Assume we have some data of measurements

SeedRandom[9];
data = Table[{i,2*(1 + 5*RandomReal[])*Exp[-0.01*(1 + RandomReal[])*i]}, {i, {0, 100, 200, 400, 700}}];

and measurement errors associated to it

errors = 10^# & /@ Range[0, -4, -1];

Ignoring the weights in a first step, the fitting works very well:

nlm = NonlinearModelFit[data, a*Exp[-b*x], {a, b}, x, Method -> "NMinimize"];
Show[ListPlot[data, PlotStyle -> PointSize -> Medium], 
Plot[nlm[x], {x, 0, 700}, PlotRange -> Full], PlotRange -> All]

Now, according to this how-to, we can include the measurement errors to the fitting procedure:

nlm = NonlinearModelFit[data, a*Exp[-b*x], {a, b}, x, Weights -> 1/errors^2, VarianceEstimatorFunction -> (1 &), Method -> "NMinimize"];
Show[ListPlot[data, PlotStyle -> PointSize -> Medium], 
Plot[nlm[x], {x, 0, 700}, PlotRange -> Full], PlotRange -> All]

Clearly, the obtained fit is much worse.

My questions are:

1. What is the objective function in NonlinearModelFit when introducing weights? The how-to says: "It is important to note that weights do not change the fitting or error estimates", but obviously the fit is changed. Thus, the weights have to be a part of the objective functions, which is minimized.

2. How can I obtain a better fit with introduced weights? I was playing around with the NMinimize options "NelderMead", "RandomSearch", "SimulatedAnnealing" and "DifferentialEvolution", but up to this point none of them was able to reproduce the fit obtained by ignoring the measurement errors.

Answer 1

The Weights option operates on a point-by-point basis to say how important that individual point is to the complete fit. Let's take some simple data that obviously don't fit on a line and try to fit them with a line. With equal weighting, you get a compromise:

data = {{1, 1}, {2, 2.5}, {3, 3}};
nlm = NonlinearModelFit[data, a x + b, {a, b}, x, Weights -> {1, 1, 1}]
Show[ListPlot[data], Plot[nlm[x], {x, 0, 7}]]

If you weight the second point a lot (in comparison to the other two then it moves the fit closer to the second point.

nlm2 = NonlinearModelFit[data, a x + b, {a, b}, x, Weights -> {1, 100, 1}]
Show[ListPlot[data], Plot[nlm2[x], {x, 0, 7}]]

while if you weight the 1st and 3rd more heavily, it almost ignores the second completely:

nlm3 = NonlinearModelFit[data, a x + b, {a, b}, x, Weights -> {10, 1, 10}]
Show[ListPlot[data], Plot[nlm3[x], {x, 0, 7}]]

So -- what is it actually doing? You'd have to ask someone from Wolfram to know for sure, but it seems highly likely it's something like this:

Minimize ( $w_1*(f[x]-p_1)^2 + w_2*(f[x]-p_2)^2 + w_3*(f[x]-p_3)^2 )$

where the $p_i$ are the points and the $f[x]$ is your function. The weights are then proportional to the values you give in the Weights option. For instance, by setting $w_2=100$, you have made the fit to point $p_2$ much more important as it chooses the parameters. Now obviously, this example is for a simple linear fit (here you don't need the full power of NonlinearModelFit but it would be surprising if it operated fundamentally differently for different kinds of equation.