4
$\begingroup$

I have a large data set with 150k lines and two data columns. I noticed that there is a linear trend, which I want to remove. So, I do the following:

First, fit with a linear model:

model = LinearModelFit[data, x, x]

then, I subtract the fit from the data:

data1 = data;
data1[[All, 2]] = data[[All, 2]] - model[data[[All, 1]]] + model[0];

This, however, takes unacceptable amount of time. Is there any way I can speed things up?

$\endgroup$

4 Answers 4

1
$\begingroup$

Since you only want to remove the linear trend. Just use Fit, and this is also easy to understand.(And a little faster than Mr Alpha's on my computer.)

data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] + Range[150000];
AbsoluteTiming[
  f[x_] = Fit[data, {1, x}, x];
  ans=Transpose@{data[[;; , 1]], data[[;; , 2]] - f@data[[;; , 1]]};]
(*==>{0.047003, Null}*)

Mr Alpha's code

AbsoluteTiming[{a, b} = 
  LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
  data1 = Transpose@{data[[All, 1]], 
  data[[All, 2]] - (a + b*data[[All, 1]])};]
(*==>{0.054003, Null}*)
$\endgroup$
6
  • $\begingroup$ Not really much of a difference in timing, which is expected, since both use least-squares methods underneath. On the other hand, since only the coefficients are wanted, FindFit[] is more expedient than Fit[] since the former yields the coefficients themselves, and the latter yields a function that you still have to extract coefficients from. $\endgroup$ May 10, 2013 at 2:42
  • $\begingroup$ @J.M. I'm not familiar with the methods underneath. Well, I think if dealing with more complex form, using my code, I don't need to construct the function manually in the end. And apply the fitted function to the data seems most natural to me. $\endgroup$
    – luyuwuli
    May 10, 2013 at 2:51
  • $\begingroup$ In that case, you can skip the use of f: Function[x, Fit[data, {1, x}, x] // Evaluate] @ data[[;; , 1]]. $\endgroup$ May 10, 2013 at 2:54
  • $\begingroup$ @J.M. Yes. From the other side, this one-line-code makes it slightly difficult to understand. (Just a matter of personal taste:) $\endgroup$
    – luyuwuli
    May 10, 2013 at 3:01
  • $\begingroup$ I like this solution the most, because it is faster and easier to understand than the others. PS. I also think FindFit is better. $\endgroup$ May 10, 2013 at 8:45
7
$\begingroup$

If all you want is to remove a linear trend from the data you don't need all the fancy statistics done by LinearModelFit and a faster alternative is to just use LeastSquares and then use the resulting parameters to remove the trend from the data.

(*Generate 150k datapoints with a linear trend*)
data = RandomVariate[NormalDistribution[0, 50], {150000, 2}] + 
   Range[150000];

(*The LinearModelFit version*)
AbsoluteTiming[
 model = LinearModelFit[data, x, x];
 data1 = data;
 data1[[All, 2]] = data[[All, 2]] - model /@ data[[All, 1]] + model[0];
 ]

(*==> {30.053463, Null} *)

(*Faster alternative*)
AbsoluteTiming[{a, b} = 
  LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]];
 data1 = Transpose@{data[[All, 1]], 
    data[[All, 2]] - (a + b*data[[All, 1]])};
 ]

(*==> {0.044045, Null} *)

It is around 700 times faster on my machine.

$\endgroup$
1
  • $\begingroup$ wow, that is a lot faster. thanks $\endgroup$ May 9, 2013 at 22:54
3
$\begingroup$

The syntax you are using is incorrect. Try model[{1,2,3}] and notice that it can't be applied to a list. Just change model[data[[All,1]]] to model /@ data[[All,1]].

This will finish in time, but it won't be fast at all (I do not know why).

This will be much faster (in place of model /@ data[[All,1]]):

model["BestFit"] /. x -> data[[All, 1]]
$\endgroup$
2
  • $\begingroup$ I do wonder why something as simple as model["BestFit"] takes a noticeable amount of time, though. It should be instantaneous. $\endgroup$
    – Szabolcs
    May 9, 2013 at 20:54
  • $\begingroup$ Thanks, but I have to note one thing. For small list model["BestFit"] /. x -> data[[All, 1]] indeed gives faster results. However, for my dataset, I had to abort, because after 5 mins it wasnt ready and had consumed 15 GB RAM. On the other hand model /@ data[[All,1]]` took only 110 sec and negligible amount of RAM $\endgroup$ May 9, 2013 at 21:35
1
$\begingroup$

If one absolutely insists on using LinearModelFit[] for linear detrending:

model = LinearModelFit[data, x, x];
trendFree = Transpose[{data[[All, 1]], model["FitResiduals"] + model[0]}];

Otherwise, we can do something quite similar to Mr. Alpha's procedure:

b = Last[LeastSquares[DesignMatrix[data, x, x], data[[All, 2]]]];
trendFree = data.{{1, -b}, {0, 1}};
$\endgroup$
3
  • $\begingroup$ No, I don't insist on LinearModelFit[]. It was the first command google gave me for fitting linear functions $\endgroup$ May 10, 2013 at 9:09
  • $\begingroup$ Hi, @phidelio. That sentence was more figurative than literal; I meant that if one is going to be using that function for detrending and other purposes besides, then at least the detrending part is easy. $\endgroup$ May 10, 2013 at 9:11
  • $\begingroup$ Useful. Been using LinearModelFit forever without concerns for performance. $\endgroup$ Apr 30, 2015 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.