I have to compute the covariance of 50 very large integer matrices (2500x2000 elements). However, according to my estimation this will take around 10 days. Do you have any ideas how to speed things up?

  • $\begingroup$ A 5000x5000 matrix calculates under 10s for me Covariance[RandomReal[1, {5000, 5000}]]; // AbsoluteTiming What type of elements do you have? $\endgroup$ – ssch Nov 14 '13 at 20:58
  • $\begingroup$ How did you estimate it will takes 10 days per matrix? (btw, 10 days using which type of computer? since the CPU type and amount of RAM should make some difference) $\endgroup$ – Nasser Nov 14 '13 at 21:01
  • $\begingroup$ @Nasser Sorry, I made a mistake; it's 10 days altogether. I compute the covariance of 100 smaller, and scaled everything assuming linearity. $\endgroup$ – phidelio Nov 14 '13 at 21:10
  • 1
    $\begingroup$ @phidelio Make sure your matrix contains only machine precision numbers, i.e. use N[matrix] instead of matrix. $\endgroup$ – Szabolcs Nov 14 '13 at 21:26
  • 1
    $\begingroup$ @phidelio Then you have your solution---use N on them. Otherwise Mathematica will attempt to calculate an exact solution, which is very slow and likely not what you need. $\endgroup$ – Szabolcs Nov 14 '13 at 21:36

If the matrix contains exact integers, Mathematica will compute an exact result (in terms of exact rational numbers). This is very slow.

If you convert your matrix to (inexact) machine precision numbers, the calculation will be much much faster.

Use Covariance@N[matrix] instead of Covariance[matrix].

  • $\begingroup$ Thanks, I will try it tomorrow and let you know (I don't have Mathematica on this computer). $\endgroup$ – phidelio Nov 14 '13 at 22:14
  • $\begingroup$ This definitely works. Thanks a lot. $\endgroup$ – phidelio Nov 15 '13 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.