0
$\begingroup$

I'm trying to create a table of the form

n=20;
V = Table[
   0, {b1, 1, n}, {a1, 1, n}, {b2, 1, n}, {a2, 1, n}, {b3, 1, n}, {a3,
     1, n}, {b4, 1, n}, {a4, 1, n}];

but it fails with a beep after some time. Is there anyway to work with large arrays? I would like to use n>100 and go up to b100, a100 or as close as possible.

$\endgroup$
1
  • 4
    $\begingroup$ Please consider ConstantArray[0, {n, n, n, n, n, n, n, n}, SparseArray] which won't consume vast amounts of memory for just storing zeroes. $\endgroup$
    – kirma
    Aug 11, 2022 at 18:36

1 Answer 1

2
$\begingroup$

Introductory example. Consider a $20 \times 20 \times 20 \times 20$ array:

A = Table[0,{20},{20},{20},{20}];

How much memory does it use?

ByteCount[A]
(* 1280224 *)

We see that it uses a little more than 1 megabyte. It is easy to understand where this number comes from: On a 64 bit computer, one number uses 8 byte, and therefore $20^4$ numbers use $20^4 \cdot 8 = 1280000$ bytes. The extra $224$ byte can be thought of as a bookkeeping overhead.

Generalization. An array with dimensions $$ \underbrace{n \times \cdots \times n}_{d} $$ will use $n^d \cdot 8$ bytes. Let us consider two examples:

  • In the question, OP uses $n=20$ and $d=8$, which gives $20^8 \cdot 8 = 204800000000$ bytes which is 205 gigabyte. That is not an outrageous amount of memory, but more than the memory of a typical personal computer. Hence, unsurprisingly, the request is not carried out.

  • OP expresses interest in $n=100$ and $d = 200$, which gives $100^{200} \cdot 8 \approx 10^{400}$ bytes and which is an outrageous amount of memory. This is not realistic.


Disclaimer. The counting I used above is simplified and assumes that the array is stored in a compact form. In Mathematica this is known as a "packed array". The array A in the introductory example is packed, as can be checked using

A // Developer`PackedArrayQ
(* True *)

Here is an example that is not packed and uses more memory than the simplified counting would suggest:

Table[i^j*k^l,{i,1,20},{j,1,20},{k,1,20},{l,1,20}]//ByteCount
(* 7045544 *)

The reason is that individual entries of this array are large and use up more than 8 byte.

$\endgroup$
1
  • $\begingroup$ Mathematica also has SparseArrays which cut storing default values (usually zeroes), but it is a bit unclear what use this $100^8$ array would actually have - for very sparse data structures, one could also use memoization, for instance. I don't think there even is a shared-memory HPC system with an order of 100 petabytes of efficiently addressable memory which would be required for non-sparse approach. $\endgroup$
    – kirma
    Aug 12, 2022 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.