# Remove linear trend from data

I have a large data set with 150k lines and two data columns. I noticed that there is a linear trend, which I want to remove. So, I do the following:

First, fit with a linear model:

model = LinearModelFit[data, x, x]


then, I subtract the fit from the data:

data1 = data;
data1[[All, 2]] = data[[All, 2]] - model[data[[All, 1]]] + model;


This, however, takes unacceptable amount of time. Is there any way I can speed things up?

Since you only want to remove the linear trend. Just use Fit, and this is also easy to understand.(And a little faster than Mr Alpha's on my computer.)

data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] + Range;
AbsoluteTiming[
f[x_] = Fit[data, {1, x}, x];
ans=Transpose@{data[[;; , 1]], data[[;; , 2]] - f@data[[;; , 1]]};]
(*==>{0.047003, Null}*)


Mr Alpha's code

AbsoluteTiming[{a, b} =
LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
data1 = Transpose@{data[[All, 1]],
data[[All, 2]] - (a + b*data[[All, 1]])};]
(*==>{0.054003, Null}*)

• Not really much of a difference in timing, which is expected, since both use least-squares methods underneath. On the other hand, since only the coefficients are wanted, FindFit[] is more expedient than Fit[] since the former yields the coefficients themselves, and the latter yields a function that you still have to extract coefficients from. – J. M. will be back soon May 10 '13 at 2:42
• @J.M. I'm not familiar with the methods underneath. Well, I think if dealing with more complex form, using my code, I don't need to construct the function manually in the end. And apply the fitted function to the data seems most natural to me. – luyuwuli May 10 '13 at 2:51
• In that case, you can skip the use of f: Function[x, Fit[data, {1, x}, x] // Evaluate] @ data[[;; , 1]]. – J. M. will be back soon May 10 '13 at 2:54
• @J.M. Yes. From the other side, this one-line-code makes it slightly difficult to understand. (Just a matter of personal taste:) – luyuwuli May 10 '13 at 3:01
• I like this solution the most, because it is faster and easier to understand than the others. PS. I also think FindFit is better. – phidelio May 10 '13 at 8:45

If all you want is to remove a linear trend from the data you don't need all the fancy statistics done by LinearModelFit and a faster alternative is to just use LeastSquares and then use the resulting parameters to remove the trend from the data.

(*Generate 150k datapoints with a linear trend*)
data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] +
Range;

(*The LinearModelFit version*)
AbsoluteTiming[
model = LinearModelFit[data, x, x];
data1 = data;
data1[[All, 2]] = data[[All, 2]] - model /@ data[[All, 1]] + model;
]

(*==> {30.053463, Null} *)

(*Faster alternative*)
AbsoluteTiming[{a, b} =
LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
data1 = Transpose@{data[[All, 1]],
data[[All, 2]] - (a + b*data[[All, 1]])};
]

(*==> {0.044045, Null} *)


It is around 700 times faster on my machine.

• wow, that is a lot faster. thanks – phidelio May 9 '13 at 22:54

The syntax you are using is incorrect. Try model[{1,2,3}] and notice that it can't be applied to a list. Just change model[data[[All,1]]] to model /@ data[[All,1]].

This will finish in time, but it won't be fast at all (I do not know why).

This will be much faster (in place of model /@ data[[All,1]]):

model["BestFit"] /. x -> data[[All, 1]]

• I do wonder why something as simple as model["BestFit"] takes a noticeable amount of time, though. It should be instantaneous. – Szabolcs May 9 '13 at 20:54
• Thanks, but I have to note one thing. For small list model["BestFit"] /. x -> data[[All, 1]] indeed gives faster results. However, for my dataset, I had to abort, because after 5 mins it wasnt ready and had consumed 15 GB RAM. On the other hand model /@ data[[All,1]] took only 110 sec and negligible amount of RAM – phidelio May 9 '13 at 21:35

If one absolutely insists on using LinearModelFit[] for linear detrending:

model = LinearModelFit[data, x, x];
trendFree = Transpose[{data[[All, 1]], model["FitResiduals"] + model}];


Otherwise, we can do something quite similar to Mr. Alpha's procedure:

b = Last[LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]]];
trendFree = data.{{1, -b}, {0, 1}};

• No, I don't insist on LinearModelFit[]`. It was the first command google gave me for fitting linear functions – phidelio May 10 '13 at 9:09
• Hi, @phidelio. That sentence was more figurative than literal; I meant that if one is going to be using that function for detrending and other purposes besides, then at least the detrending part is easy. – J. M. will be back soon May 10 '13 at 9:11
• Useful. Been using LinearModelFit forever without concerns for performance. – John Morganthau Apr 30 '15 at 19:21