Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a large data set with 150k lines and two data columns. I noticed that there is a linear trend, which I want to remove. So, I do the following:

first fit with a linear model

model = LinearModelFit[data, x, x]

then, I substract the fit from the data

data1 = data;
data1[[All, 2]] = data[[All, 2]] - model[data[[All, 1]]] + model[0];

This, however, takes unacceptable amount of time. Is there any way I can speed things up.

share|improve this question

4 Answers 4

up vote 1 down vote accepted

Since you only want to remove the linear trend. Just use Fit, and this is also easy to understand.(And a little faster than Mr Alpha's on my computer.)

data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] + Range[150000];
AbsoluteTiming[
  f[x_] = Fit[data, {1, x}, x];
  ans=Transpose@{data[[;; , 1]], data[[;; , 2]] - f@data[[;; , 1]]};]
(*==>{0.047003, Null}*)

Mr Alpha's code

AbsoluteTiming[{a, b} = 
  LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
  data1 = Transpose@{data[[All, 1]], 
  data[[All, 2]] - (a + b*data[[All, 1]])};]
(*==>{0.054003, Null}*)
share|improve this answer
    
Not really much of a difference in timing, which is expected, since both use least-squares methods underneath. On the other hand, since only the coefficients are wanted, FindFit[] is more expedient than Fit[] since the former yields the coefficients themselves, and the latter yields a function that you still have to extract coefficients from. –  J. M. May 10 '13 at 2:42
    
@J.M. I'm not familiar with the methods underneath. Well, I think if dealing with more complex form, using my code, I don't need to construct the function manually in the end. And apply the fitted function to the data seems most natural to me. –  luyuwuli May 10 '13 at 2:51
    
In that case, you can skip the use of f: Function[x, Fit[data, {1, x}, x] // Evaluate] @ data[[;; , 1]]. –  J. M. May 10 '13 at 2:54
    
@J.M. Yes. From the other side, this one-line-code makes it slightly difficult to understand. (Just a matter of personal taste:) –  luyuwuli May 10 '13 at 3:01
    
I like this solution the most, because it is faster and easier to understand than the others. PS. I also think FindFit is better. –  phidelio May 10 '13 at 8:45

If all you want is to remove a linear trend from the data you don't need all the fancy statistics done by LinearModelFit and a faster alternative is to just use LeastSquares and then use the resulting parameters to remove the trend from the data.

(*Generate 150k datapoints with a linear trend*)
data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] + 
   Range[150000];

(*The LinearModelFit version*)
AbsoluteTiming[
 model = LinearModelFit[data, x, x];
 data1 = data;
 data1[[All, 2]] = data[[All, 2]] - model /@ data[[All, 1]] + model[0];
 ]

(*==> {30.053463, Null} *)

(*Faster alternative*)
AbsoluteTiming[{a, b} = 
  LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
 data1 = Transpose@{data[[All, 1]], 
    data[[All, 2]] - (a + b*data[[All, 1]])};
 ]

(*==> {0.044045, Null} *)

It is around 700 times faster on my machine.

share|improve this answer
    
wow, that is a lot faster. thanks –  phidelio May 9 '13 at 22:54

The syntax you are using is incorrect. Try model[{1,2,3}] and notice that it can't be applied to a list. Just change model[data[[All,1]]] to model /@ data[[All,1]].

This will finish in time, but it won't be fast at all (I do not know why).

This will be much faster (in place of model /@ data[[All,1]]):

model["BestFit"] /. x -> data[[All, 1]]
share|improve this answer
    
I do wonder why something as simple as model["BestFit"] takes a noticeable amount of time, though. It should be instantaneous. –  Szabolcs May 9 '13 at 20:54
    
Thanks, but I have to note one thing. For small list model["BestFit"] /. x -> data[[All, 1]] indeed gives faster results. However, for my dataset, I had to abort, because after 5 mins it wasnt ready and had consumed 15 GB RAM. On the other hand model /@ data[[All,1]]` took only 110 sec and negligible amount of RAM –  phidelio May 9 '13 at 21:35

If one absolutely insists on using LinearModelFit[] for linear detrending:

model = LinearModelFit[data, x, x];
trendFree = Transpose[{data[[All, 1]], model["FitResiduals"] + model[0]}];

Otherwise, we can do something quite similar to Mr. Alpha's procedure:

b = Last[LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]]];
trendFree = data.{{1, -b}, {0, 1}};
share|improve this answer
    
No, I dont insist on LinearModelFit[]`. It was the first command google gave me for fitting linear functions –  phidelio May 10 '13 at 9:09
    
Hi, @phidelio. That sentence was more figurative than literal; I meant that if one is going to be using that function for detrending and other purposes besides, then at least the detrending part is easy. –  J. M. May 10 '13 at 9:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.