I have an array of 800,000 data measured at 1.1s time gap. It is oscillating data but has clear linear drift. I want to remove this linear drift. But the estimates by FindFit and Fit has been very bad.
Here is my attempt with FindFit

model = e*x + noise[[1]];  
fit = FindFit[noise, model, {e}, x]
{e -> -0.0430509}

The value of e is quite visibly wrong. It should be arround -0.040. Is there better ways to find and remove linear trends from datas? Thank you very much.

The data can be found here: https://www.dropbox.com/s/xg2w6d4a6bpaejk/data.dat

  • $\begingroup$ Can you provide the data set ? $\endgroup$ – Sektor Aug 14 '14 at 11:33
  • $\begingroup$ @Sektor I have updated the link to the data. $\endgroup$ – jason Aug 14 '14 at 11:48
  • $\begingroup$ I am already downloading it :) $\endgroup$ – Sektor Aug 14 '14 at 11:48
  • $\begingroup$ Please let me know, if you have a better way. $\endgroup$ – jason Aug 14 '14 at 11:58
  • $\begingroup$ Can you try the following ? swd = StationaryWaveletTransform[data, DaubechiesWavelet[2], 8] and then ListLogLogPlot[ data - .93 Flatten@Reverse@swd[{0, 0, 0, 0, 0, 0, 0, 1}, "Values"]] $\endgroup$ – Sektor Aug 14 '14 at 13:10

Fit seems to work fine, you just need to include your 1.1 step size.

data = Import["data.dat", "List"];
d = Thread[{1.1 Range@Length@d, d}];
Fit[d, {1, x}, x]

8.24575*10^8 - 0.0402358 x

So now we have the -0.040 that you wanted. It looks ok by eye:

Show[ListPlot[d, Joined -> True], 
     Plot[8.245747660409383`*^8 - 0.04023578596262912` x, {x, 1.1, 1.1 Length@d},
       PlotStyle -> Red]]


We can now subtract the linear drift from the data:

dflat = {First@#, Last@# - (8.245747660409383`*^8 - 0.04023578596262912` First@#)}& /@ d;
ListPlot[dflat, Joined -> True]

enter image description here

Edit: We can take this even further. Let's interpolate over every 5000th point:

i = Interpolation[dflat[[;; ;; 5000]]];
Show[ListPlot[dflat, Joined -> True], 
     Plot[i[x], {x, dflat[[1, 1]], dflat[[-1, 1]]}, PlotStyle -> Red]]

enter image description here

Then subtract this from the data:

dsharp = Quiet@{First@#, Last@# - i[First@#]} & /@ dflat;
ListPlot[dsharp, Joined -> True, PlotRange -> All]

enter image description here

Now we're really just oscillating around a fixed value. But I have no idea what your data represents. Maybe this is what you want, or maybe this is just the noise.

| improve this answer | |
  • $\begingroup$ Thank you very much this helps a lot. The final result is what I was looking for too. :) $\endgroup$ – jason Aug 18 '14 at 18:07
  • $\begingroup$ @jason Then as a further though, you might not want to just take every 5000th point raw for the interpolation. You might want to smooth it our first. MedianFilter could be a good choice. $\endgroup$ – wxffles Aug 18 '14 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.