# Removing Linear drift from an oscillating data

I have an array of 800,000 data measured at 1.1s time gap. It is oscillating data but has clear linear drift. I want to remove this linear drift. But the estimates by FindFit and Fit has been very bad.
Here is my attempt with FindFit

model = e*x + noise[[1]];
fit = FindFit[noise, model, {e}, x]
{e -> -0.0430509}

The value of e is quite visibly wrong. It should be arround -0.040. Is there better ways to find and remove linear trends from datas? Thank you very much.

The data can be found here: https://www.dropbox.com/s/xg2w6d4a6bpaejk/data.dat

-
Can you provide the data set ? – Sektor Aug 14 '14 at 11:33
@Sektor I have updated the link to the data. – jason Aug 14 '14 at 11:48
Please let me know, if you have a better way. – jason Aug 14 '14 at 11:58
Can you try the following ? swd = StationaryWaveletTransform[data, DaubechiesWavelet[2], 8] and then ListLogLogPlot[ data - .93 Flatten@Reverse@swd[{0, 0, 0, 0, 0, 0, 0, 1}, "Values"]] – Sektor Aug 14 '14 at 13:10

Fit seems to work fine, you just need to include your 1.1 step size.

data = Import["data.dat", "List"];
Fit[d, {1, x}, x]

8.24575*10^8 - 0.0402358 x

So now we have the -0.040 that you wanted. It looks ok by eye:

Show[ListPlot[d, Joined -> True],
Plot[8.245747660409383*^8 - 0.04023578596262912 x, {x, 1.1, 1.1 Length@d},
PlotStyle -> Red]]

We can now subtract the linear drift from the data:

dflat = {First@#, Last@# - (8.245747660409383*^8 - 0.04023578596262912 First@#)}& /@ d;
ListPlot[dflat, Joined -> True]

Edit: We can take this even further. Let's interpolate over every 5000th point:

i = Interpolation[dflat[[;; ;; 5000]]];
Show[ListPlot[dflat, Joined -> True],
Plot[i[x], {x, dflat[[1, 1]], dflat[[-1, 1]]}, PlotStyle -> Red]]

Then subtract this from the data:

dsharp = Quiet@{First@#, Last@# - i[First@#]} & /@ dflat;
ListPlot[dsharp, Joined -> True, PlotRange -> All]

Now we're really just oscillating around a fixed value. But I have no idea what your data represents. Maybe this is what you want, or maybe this is just the noise.

-
Thank you very much this helps a lot. The final result is what I was looking for too. :) – jason Aug 18 '14 at 18:07
@jason Then as a further though, you might not want to just take every 5000th point raw for the interpolation. You might want to smooth it our first. MedianFilter could be a good choice. – wxffles Aug 18 '14 at 22:24