7
$\begingroup$

I have need to store large square matrices in memory, some times sparse, some times not. I have used up to $2^{15}$ by $2^{15}$ as my upper bound so far, and Mathematica has handled it even if it took quite a while :).

I would like to try to do some calculations on a $2^{20}$ by $2^{20}$ now, but it will take quite a long time to compute... so I need to know in advance if Mathematica will be able to handle that much memory use. Answers for both sparse and dense cases are encouraged.

Improve this question
$\endgroup$
7
  • 8
    $\begingroup$ A $2^{15}$ dense square matrix takes 8GB of RAM. A $2^{20}$ one takes 8TB. Do you have that much RAM? I think Mathematica should be able to handle it, if your OS can. However, I've never tested this, as the biggest computer I have access to has only 4TB of RAM, and Mathematica isn't available on it. $\endgroup$
    – Oleksandr R.
    Jan 26, 2015 at 18:51
  • 2
    $\begingroup$ @OleksandrR. where do you get a computer that has that much physical RAM? I want one, or two ... dozen. $\endgroup$
    – rcollyer
    Jan 26, 2015 at 19:44
  • $\begingroup$ @rcollyer the cheapest way is probably to get 32 computers each with 256GB of RAM, connect them together using 40Gbit/s Infiniband, and install vSMP on them. Alternatively you could buy an SGI UV. $\endgroup$
    – Oleksandr R.
    Jan 26, 2015 at 19:50
  • 1
    $\begingroup$ @OleksandrR. here I was hoping you could get a single motherboard with that much RAM, but alas, 512GB seems to be the limit. $\endgroup$
    – rcollyer
    Jan 26, 2015 at 19:58
  • 2
    $\begingroup$ The other thing is, many matrix operations have a complexity of between $O(N^2)$ and $O(N^3)$. So, be prepared for your calculation with the $2^{20}$ matrix to take several (tens of) thousands of times longer than "quite a while". $\endgroup$
    – Oleksandr R.
    Jan 26, 2015 at 22:56

1 Answer 1

Reset to default
9
$\begingroup$

Oleksandr remarked on the memory required for a dense matrix. I shall attempt to explore SparseArray limitations.

From this error message it appears that the dimensions of the array must be machine integers:

SparseArray[{}, {2, 2}^70]

SparseArray::adims: Array dimension specification {1180591620717411303424,1180591620717411303424} should be Automatic, a non-negative machine integer, or a list of non-negative machine integers. >>

If smaller dimensions are used it may still take too much memory:

SparseArray[{}, {2, 2}^60]

No more memory available.
Mathematica kernel has shut down.
Try quitting other applications and then retry.

From earlier experimentation I determined that every row in the arrays takes significant memory, so if we limit the number of rows:

SparseArray[{}, {2^20, 2^63 - 2}] // ByteCount

8389184

Mathematica is easily able to address a sparse array with 2^20 rows and on my machine up to 2^63 - 2 columns. However transposing this array would be impossible.

While we can assemble a SparseArray of this size with some non-zero elements:

rls = 
 Array[
   Round[RandomReal[1, 2]*{2^20, 2^63 - 2}] -> # &,
   1000
 ];

test = SparseArray[rls, {2^20, 2^63 - 2}];

and some fundamental operations are possible:

3*test // ByteCount

8405720

others operations are not:

SparseArray[{1 -> 2, 7 -> 14, 183 -> 99}, 2^20].test

No more memory available.
Mathematica kernel has shut down.
Try quitting other applications and then retry.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.