Table size/dimension error

Question

I'm trying to create a table of the form

n=20;
V = Table[
   0, {b1, 1, n}, {a1, 1, n}, {b2, 1, n}, {a2, 1, n}, {b3, 1, n}, {a3,
     1, n}, {b4, 1, n}, {a4, 1, n}];

but it fails with a beep after some time. Is there anyway to work with large arrays? I would like to use n>100 and go up to b100, a100 or as close as possible.

Answer 1

Introductory example. Consider a $20 \times 20 \times 20 \times 20$ array:

A = Table[0,{20},{20},{20},{20}];

How much memory does it use?

ByteCount[A]
(* 1280224 *)

We see that it uses a little more than 1 megabyte. It is easy to understand where this number comes from: On a 64 bit computer, one number uses 8 byte, and therefore $20^4$ numbers use $20^4 \cdot 8 = 1280000$ bytes. The extra $224$ byte can be thought of as a bookkeeping overhead.

Generalization. An array with dimensions $$ \underbrace{n \times \cdots \times n}_{d} $$ will use $n^d \cdot 8$ bytes. Let us consider two examples:

• In the question, OP uses $n=20$ and $d=8$, which gives $20^8 \cdot 8 = 204800000000$ bytes which is 205 gigabyte. That is not an outrageous amount of memory, but more than the memory of a typical personal computer. Hence, unsurprisingly, the request is not carried out.

• OP expresses interest in $n=100$ and $d = 200$, which gives $100^{200} \cdot 8 \approx 10^{400}$ bytes and which is an outrageous amount of memory. This is not realistic.

---

Disclaimer. The counting I used above is simplified and assumes that the array is stored in a compact form. In Mathematica this is known as a "packed array". The array A in the introductory example is packed, as can be checked using

A // Developer`PackedArrayQ
(* True *)

Here is an example that is not packed and uses more memory than the simplified counting would suggest:

Table[i^j*k^l,{i,1,20},{j,1,20},{k,1,20},{l,1,20}]//ByteCount
(* 7045544 *)

The reason is that individual entries of this array are large and use up more than 8 byte.