# Remove linear trend from data

I have a large data set with 150k lines and two data columns. I noticed that there is a linear trend, which I want to remove. So, I do the following:

first fit with a linear model

model = LinearModelFit[data, x, x]


then, I substract the fit from the data

data1 = data;
data1[[All, 2]] = data[[All, 2]] - model[data[[All, 1]]] + model[0];


This, however, takes unacceptable amount of time. Is there any way I can speed things up.

-

Since you only want to remove the linear trend. Just use Fit, and this is also easy to understand.(And a little faster than Mr Alpha's on my computer.)

data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] + Range[150000];
AbsoluteTiming[
f[x_] = Fit[data, {1, x}, x];
ans=Transpose@{data[[;; , 1]], data[[;; , 2]] - f@data[[;; , 1]]};]
(*==>{0.047003, Null}*)


Mr Alpha's code

AbsoluteTiming[{a, b} =
LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
data1 = Transpose@{data[[All, 1]],
data[[All, 2]] - (a + b*data[[All, 1]])};]
(*==>{0.054003, Null}*)

-
Not really much of a difference in timing, which is expected, since both use least-squares methods underneath. On the other hand, since only the coefficients are wanted, FindFit[] is more expedient than Fit[] since the former yields the coefficients themselves, and the latter yields a function that you still have to extract coefficients from. –  Ｊ. Ｍ. May 10 at 2:42
@J.M. I'm not familiar with the methods underneath. Well, I think if dealing with more complex form, using my code, I don't need to construct the function manually in the end. And apply the fitted function to the data seems most natural to me. –  luyuwuli May 10 at 2:51
In that case, you can skip the use of f: Function[x, Fit[data, {1, x}, x] // Evaluate] @ data[[;; , 1]]. –  Ｊ. Ｍ. May 10 at 2:54
@J.M. Yes. From the other side, this one-line-code makes it slightly difficult to understand. (Just a matter of personal taste:) –  luyuwuli May 10 at 3:01
I like this solution the most, because it is faster and easier to understand than the others. PS. I also think FindFit is better. –  phidelio May 10 at 8:45
show 1 more comment

If all you want is to remove a linear trend from the data you don't need all the fancy statistics done by LinearModelFit and a faster alternative is to just use LeastSquares and then use the resulting parameters to remove the trend from the data.

(*Generate 150k datapoints with a linear trend*)
data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] +
Range[150000];

(*The LinearModelFit version*)
AbsoluteTiming[
model = LinearModelFit[data, x, x];
data1 = data;
data1[[All, 2]] = data[[All, 2]] - model /@ data[[All, 1]] + model[0];
]

(*==> {30.053463, Null} *)

(*Faster alternative*)
AbsoluteTiming[{a, b} =
LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
data1 = Transpose@{data[[All, 1]],
data[[All, 2]] - (a + b*data[[All, 1]])};
]

(*==> {0.044045, Null} *)


It is around 700 times faster on my machine.

-
wow, that is a lot faster. thanks –  phidelio May 9 at 22:54

The syntax you are using is incorrect. Try model[{1,2,3}] and notice that it can't be applied to a list. Just change model[data[[All,1]]] to model /@ data[[All,1]].

This will finish in time, but it won't be fast at all (I do not know why).

This will be much faster (in place of model /@ data[[All,1]]):

model["BestFit"] /. x -> data[[All, 1]]

-
I do wonder why something as simple as model["BestFit"] takes a noticeable amount of time, though. It should be instantaneous. –  Szabolcs May 9 at 20:54
Thanks, but I have to note one thing. For small list model["BestFit"] /. x -> data[[All, 1]] indeed gives faster results. However, for my dataset, I had to abort, because after 5 mins it wasnt ready and had consumed 15 GB RAM. On the other hand model /@ data[[All,1]] took only 110 sec and negligible amount of RAM –  phidelio May 9 at 21:35

If one absolutely insists on using LinearModelFit[] for linear detrending:

model = LinearModelFit[data, x, x];
trendFree = Transpose[{data[[All, 1]], model["FitResiduals"] + model[0]}];


Otherwise, we can do something quite similar to Mr. Alpha's procedure:

b = Last[LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]]];
trendFree = data.{{1, -b}, {0, 1}};

-
No, I dont insist on LinearModelFit[]. It was the first command google gave me for fitting linear functions –  phidelio May 10 at 9:09
Hi, @phidelio. That sentence was more figurative than literal; I meant that if one is going to be using that function for detrending and other purposes besides, then at least the detrending part is easy. –  Ｊ. Ｍ. May 10 at 9:11