Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have an array of 800,000 data measured at 1.1s time gap. It is oscillating data but has clear linear drift. I want to remove this linear drift. But the estimates by FindFit and Fit has been very bad.
Here is my attempt with FindFit

model = e*x + noise[[1]];  
fit = FindFit[noise, model, {e}, x]
{e -> -0.0430509}

The value of e is quite visibly wrong. It should be arround -0.040. Is there better ways to find and remove linear trends from datas? Thank you very much.

The data can be found here: https://www.dropbox.com/s/xg2w6d4a6bpaejk/data.dat

share|improve this question
    
Can you provide the data set ? –  Sektor Aug 14 at 11:33
    
@Sektor I have updated the link to the data. –  jason Aug 14 at 11:48
    
I am already downloading it :) –  Sektor Aug 14 at 11:48
    
Please let me know, if you have a better way. –  jason Aug 14 at 11:58
    
Can you try the following ? swd = StationaryWaveletTransform[data, DaubechiesWavelet[2], 8] and then ListLogLogPlot[ data - .93 Flatten@Reverse@swd[{0, 0, 0, 0, 0, 0, 0, 1}, "Values"]] –  Sektor Aug 14 at 13:10

1 Answer 1

up vote 6 down vote accepted

Fit seems to work fine, you just need to include your 1.1 step size.

data = Import["data.dat", "List"];
d = Thread[{1.1 Range@Length@d, d}];
Fit[d, {1, x}, x]

8.24575*10^8 - 0.0402358 x

So now we have the -0.040 that you wanted. It looks ok by eye:

Show[ListPlot[d, Joined -> True], 
     Plot[8.245747660409383`*^8 - 0.04023578596262912` x, {x, 1.1, 1.1 Length@d},
       PlotStyle -> Red]]

fit

We can now subtract the linear drift from the data:

dflat = {First@#, Last@# - (8.245747660409383`*^8 - 0.04023578596262912` First@#)}& /@ d;
ListPlot[dflat, Joined -> True]

enter image description here

Edit: We can take this even further. Let's interpolate over every 5000th point:

i = Interpolation[dflat[[;; ;; 5000]]];
Show[ListPlot[dflat, Joined -> True], 
     Plot[i[x], {x, dflat[[1, 1]], dflat[[-1, 1]]}, PlotStyle -> Red]]

enter image description here

Then subtract this from the data:

dsharp = Quiet@{First@#, Last@# - i[First@#]} & /@ dflat;
ListPlot[dsharp, Joined -> True, PlotRange -> All]

enter image description here

Now we're really just oscillating around a fixed value. But I have no idea what your data represents. Maybe this is what you want, or maybe this is just the noise.

share|improve this answer
    
Thank you very much this helps a lot. The final result is what I was looking for too. :) –  jason Aug 18 at 18:07
    
@jason Then as a further though, you might not want to just take every 5000th point raw for the interpolation. You might want to smooth it our first. MedianFilter could be a good choice. –  wxffles Aug 18 at 22:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.