I am attempting to use the least squares method to fit a polynomial to a gapped dataset. Regularization was performed assuming variances defined by the function:

$$v_{\epsilon_n} = 0.01 + 0.001(t_n + t_0).$$

There is a large period of time where the recording instrument was offline and no data were recorded, which seems to decrease the quality of my fit.

Adding weights slightly improves my result; however, the result is still poor. Looking at the shape of the data, it would seem a 5th-order polynomial would be more than sufficient. This leads me to think that I am not handing this gap correctly (currently I'm doing nothing about it ...).

My first thought was to split the dataset and produce two separate fits that I could then combine, but this seems unnecessary and messy. Is there an elegant solution to handling this gap?

Data (grey dots), 5th-order polynomial fit (blue), and 5th-order polynomial fit with weights (orange)

For reference, here is how I've computed these solutions:

data = Import["...~/filepath/practical.dat"];
t = data[[All, 1]];
d = data[[All, 2]];
ts = t/(Last[t] - t[[1]]) - 15.3;
k = 6;
n = Length[t];

a = Table[ts[[i]]^(j - 1), {i, n}, {j, k}];
c = DiagonalMatrix[Array[(1/(0.01 + (t[[#]] - t[[1]])*.001)) &, n]];
cs = c/(Last[t] - t[[1]]);
m = PseudoInverse[a\[Transpose].a].a\[Transpose].d;
mw = PseudoInverse[a\[Transpose].c.a].a\[Transpose].c.d;
soln = MapThread[{#1, #2} &, {t, a.m}];
wsoln = MapThread[{#1, #2} &, {t, a.mw}];

Using Fit, I can get the same unweighted result (blue curve), so at least I know my least squares soln is correct ...


  • 2
    $\begingroup$ The gap is not your problem. It is attempting to fit a polynomial to the data. (Although you can do much better with k=9.) You'll get a much better fit (although the change in variance is not accounted for) if you use smoothing splines (mathematica.stackexchange.com/questions/33206/…) or Anton Antonov's quantile regression package (mathematicaforprediction.wordpress.com/2014/04/19/…). $\endgroup$ – JimB Oct 18 '17 at 20:35
  • 1
    $\begingroup$ It looks a lot like a slowly growing sinusoid... why not use that instead of a polynomial? $\endgroup$ – bill s Oct 18 '17 at 21:30
  • $\begingroup$ This is just an exercise where I am required to fit a 5th-order polynomial using least squares. Basically, I was under the impression that something was wrong because multiple people (professors actually) told me that this fit wasn't good enough... $\endgroup$ – acquisto2 Oct 19 '17 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.