# Removing Linear drift from an oscillating data

I have an array of 800,000 data measured at 1.1s time gap. It is oscillating data but has clear linear drift. I want to remove this linear drift. But the estimates by FindFit and Fit has been very bad.
Here is my attempt with FindFit

model = e*x + noise[[1]];
fit = FindFit[noise, model, {e}, x]
{e -> -0.0430509}


The value of e is quite visibly wrong. It should be arround -0.040. Is there better ways to find and remove linear trends from datas? Thank you very much.

The data can be found here: https://www.dropbox.com/s/xg2w6d4a6bpaejk/data.dat

• Can you provide the data set ? Commented Aug 14, 2014 at 11:33
• @Sektor I have updated the link to the data. Commented Aug 14, 2014 at 11:48
• I am already downloading it :) Commented Aug 14, 2014 at 11:48
• Please let me know, if you have a better way. Commented Aug 14, 2014 at 11:58
• Can you try the following ? swd = StationaryWaveletTransform[data, DaubechiesWavelet[2], 8] and then ListLogLogPlot[ data - .93 Flatten@Reverse@swd[{0, 0, 0, 0, 0, 0, 0, 1}, "Values"]] Commented Aug 14, 2014 at 13:10

## 1 Answer

Fit seems to work fine, you just need to include your 1.1 step size.

data = Import["data.dat", "List"];
d = Thread[{1.1 Range@Length@d, d}];
Fit[d, {1, x}, x]


8.24575*10^8 - 0.0402358 x

So now we have the -0.040 that you wanted. It looks ok by eye:

Show[ListPlot[d, Joined -> True],
Plot[8.245747660409383*^8 - 0.04023578596262912 x, {x, 1.1, 1.1 Length@d},
PlotStyle -> Red]]


We can now subtract the linear drift from the data:

dflat = {First@#, Last@# - (8.245747660409383*^8 - 0.04023578596262912 First@#)}& /@ d;
ListPlot[dflat, Joined -> True]


Edit: We can take this even further. Let's interpolate over every 5000th point:

i = Interpolation[dflat[[;; ;; 5000]]];
Show[ListPlot[dflat, Joined -> True],
Plot[i[x], {x, dflat[[1, 1]], dflat[[-1, 1]]}, PlotStyle -> Red]]


Then subtract this from the data:

dsharp = Quiet@{First@#, Last@# - i[First@#]} & /@ dflat;
ListPlot[dsharp, Joined -> True, PlotRange -> All]


Now we're really just oscillating around a fixed value. But I have no idea what your data represents. Maybe this is what you want, or maybe this is just the noise.

• Thank you very much this helps a lot. The final result is what I was looking for too. :) Commented Aug 18, 2014 at 18:07
• @jason Then as a further though, you might not want to just take every 5000th point raw for the interpolation. You might want to smooth it our first. MedianFilter could be a good choice. Commented Aug 18, 2014 at 22:24