4
$\begingroup$

I have an array of 800,000 data measured at 1.1s time gap. It is oscillating data but has clear linear drift. I want to remove this linear drift. But the estimates by FindFit and Fit has been very bad.
Here is my attempt with FindFit

model = e*x + noise[[1]];  
fit = FindFit[noise, model, {e}, x]
{e -> -0.0430509}

The value of e is quite visibly wrong. It should be arround -0.040. Is there better ways to find and remove linear trends from datas? Thank you very much.

The data can be found here: https://www.dropbox.com/s/xg2w6d4a6bpaejk/data.dat

$\endgroup$
10
  • $\begingroup$ Can you provide the data set ? $\endgroup$
    – Sektor
    Commented Aug 14, 2014 at 11:33
  • $\begingroup$ @Sektor I have updated the link to the data. $\endgroup$
    – jason
    Commented Aug 14, 2014 at 11:48
  • $\begingroup$ I am already downloading it :) $\endgroup$
    – Sektor
    Commented Aug 14, 2014 at 11:48
  • $\begingroup$ Please let me know, if you have a better way. $\endgroup$
    – jason
    Commented Aug 14, 2014 at 11:58
  • $\begingroup$ Can you try the following ? swd = StationaryWaveletTransform[data, DaubechiesWavelet[2], 8] and then ListLogLogPlot[ data - .93 Flatten@Reverse@swd[{0, 0, 0, 0, 0, 0, 0, 1}, "Values"]] $\endgroup$
    – Sektor
    Commented Aug 14, 2014 at 13:10

1 Answer 1

7
$\begingroup$

Fit seems to work fine, you just need to include your 1.1 step size.

data = Import["data.dat", "List"];
d = Thread[{1.1 Range@Length@d, d}];
Fit[d, {1, x}, x]

8.24575*10^8 - 0.0402358 x

So now we have the -0.040 that you wanted. It looks ok by eye:

Show[ListPlot[d, Joined -> True], 
     Plot[8.245747660409383`*^8 - 0.04023578596262912` x, {x, 1.1, 1.1 Length@d},
       PlotStyle -> Red]]

fit

We can now subtract the linear drift from the data:

dflat = {First@#, Last@# - (8.245747660409383`*^8 - 0.04023578596262912` First@#)}& /@ d;
ListPlot[dflat, Joined -> True]

enter image description here

Edit: We can take this even further. Let's interpolate over every 5000th point:

i = Interpolation[dflat[[;; ;; 5000]]];
Show[ListPlot[dflat, Joined -> True], 
     Plot[i[x], {x, dflat[[1, 1]], dflat[[-1, 1]]}, PlotStyle -> Red]]

enter image description here

Then subtract this from the data:

dsharp = Quiet@{First@#, Last@# - i[First@#]} & /@ dflat;
ListPlot[dsharp, Joined -> True, PlotRange -> All]

enter image description here

Now we're really just oscillating around a fixed value. But I have no idea what your data represents. Maybe this is what you want, or maybe this is just the noise.

$\endgroup$
2
  • $\begingroup$ Thank you very much this helps a lot. The final result is what I was looking for too. :) $\endgroup$
    – jason
    Commented Aug 18, 2014 at 18:07
  • $\begingroup$ @jason Then as a further though, you might not want to just take every 5000th point raw for the interpolation. You might want to smooth it our first. MedianFilter could be a good choice. $\endgroup$
    – wxffles
    Commented Aug 18, 2014 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.