Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'd like to do a simple regression with millions of rows of (x,y) data. It seems to be too long for Mathematica. Is there a way to speed it up? It shouldn't be that hard to do, I am not sure what Mathematica is doing.

For example here's how the timing grows with the size of the input:

    First@Timing[LinearModelFit[RandomReal[NormalDistribution[], {#, 2}], x, x];] & 
        /@ {10, 100, 1000, 10000, 100000}

generating on my machine:

    {0., 0.015600, 0.046800, 0.670804, 26.722971}

It doesn't seem like it should be taking so much longer for longer data, and yet it does.

share|improve this question
2  
Why do you want to use all your (millions of) points? To get more precision? ask yourself what precision you do need, and how many points are needed to get that. I think that will be enough, but there are also several statistically interesting ways to work with subsets and then combine the results. –  belisarius Jan 15 '13 at 19:08
add comment

1 Answer

up vote 9 down vote accepted

LinearModelFit does too much--it computes residuals, etc., etc. When working with large problems, just compute what you need when you need it. It all begins with the fit itself, which should be done with LeastSquares:

First[Timing[
LeastSquares[{ConstantArray[1, #], RandomReal[NormalDistribution[], #]}\[Transpose], 
 RandomReal[NormalDistribution[], #]]]] & /@ (10^Range[7])

{0., 0., 0., 0., 0.047, 0.437, 4.508}

More than half that time is just the random number generation. In brief, computing the fit only takes about one second per four million points. The timing with more independent variables will be proportionately longer (it should scale linearly with the number of points and cubically with the number of variables).

Once datasets get this large, regression (with just a single independent variable) is usually pointless: you can do just fine by randomly subsampling the data, sometimes with as few as a hundred points or so, depending on the sizes of the residuals and how precise the estimates need to be.

share|improve this answer
    
Thanks. And if I need p-values or Rsquareds etc. should I just calculate them myself? Wish there was an option to LinearModelFit to delay computation of extraneous stuff until explicitly called for. –  Philip Maymin Jan 16 '13 at 4:01
1  
I have not found any option in LinearModelFit to delay those calculations. Fortunately, all the ancillary calculations are easy to carry out--it's just a bit of a pain. But any regression with millions of data points is going to take some work :-). (I suppose in MMA 9 you could link to R and use its lm, summary, predict, etc. functions.) –  whuber Jan 16 '13 at 14:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.