Is there a way to calculate this big set using the hard disk space and use it from there?

I aim to work with two sets, obtained through a function. One of these are:

n=3; m=3;
bigset=Take[Tuples[{1, -1}, n^m], 2^(n^m-1)];


(It calculates the set Tuples[{1, -1}, n^m] and takes the first half of it)

This set is easy to compute but hard to store (more than 100 GB, I think). I have only 8 GB of RAM. Every time I run this code, the computer freezes forever. Is there a way to calculate this big set using the hard disk space (taking no more than one night) and use it from there?

EDIT: The other set (called matrixset) can be obtained in the following way:

ClearAll["Global*"]

m = 3;
n = 3;

Inm = Tuples[Table[i, {i, 1, n}], m];
extBn = Tuples[{1, -1}, n];
prodBn = Tuples[Table[extBn, {m}]];
w[i_, j_] := Product[prodBn[[i]][[k]][[Inm[[j]][[k]]]], {k, 1, m}];
Vnm = DeleteDuplicates[
Table[Table[w[i, j], {j, 1, Length[Inm]}], {i, 1, Length[prodBn]}]];
remove = Union[Tuples[{1}, n^m], Tuples[{-1}, n^m]];
V0 = {{Vnm[[1]]}};
S = Complement[Vnm, remove]; "";
S0 = Table[{S[[i]]}, {i, 1, Length[S]}]; ""; Clear[S];
f[S0_, {i_, V0_}] := Module[{S1, S2}, S1 = Join @@@ Tuples[{S0, V0}];
S2 = Pick[DeleteDuplicatesBy[Sort]@S1,
MatrixRank[#] == i & /@ DeleteDuplicatesBy[Sort]@S1];
{i + 1, S2}]
matrixset = Last@Nest[f[S0, #] &, {2, V0}, n^m - 1]; ""; Clear[S0];


The set consist of all invertible matrices $27\times 27$ that have the unit vector $(1,...,1)$ as the last line and vectors from the set $Vnm$ as the other lines. The process remove those that differ by row-switching.

The set S0 consists of 1-row matrices from $Vnm\backslash \{(1~\cdots~1), (-1~\cdots~-1)\}$. Here is how the Module work: (1) Prepend all vectors from S0 to the 1-line matrix $(1~\cdots~1)$; (2) obtain S1 by eliminating the matrices duplicated by row-switching and those with rank $< 2$; (3) Prepend all vectors from S0 to all matrices from S1; (4) obtain S2 by eliminating the matrices duplicated by row-switching and those with rank $< 3$; (5) and so on.....

1 Answer

Observation: If you replace Tuples[{1, -1}... with Tuples[{0, 1}... you'll be generating the base-2 integer digits of the numbers 0 through 2^(n^m-1)-1.

Therefore instead of generating the entire bigset you can simply realize that

bigset[[i]] == PadLeft[IntegerDigits[i - 1, 2], n^m] /. {0 -> 1, 1 -> -1}


or as J.M. points out, a much terser and performance-wise better variant of the expression above is

1 - 2 IntegerDigits[i - 1, 2, n^m]


Observations regarding memory consumption: you are in intermediate calculations generating a 2^27 * 27 array of integers (each of which are 64 bits in size). In principle, you could manage with 32 GB RAM or maybe slightly more. But since every 27-integer tuple is 216 bytes in size, when it could be represented by a single integer you are wasting memory by a factor of 27! Now my machine has also 8GB memory and generating range = Range[0, 2^26 - 1]; is no problem whatsoever and consumes only 500-something MB of memory.

• 1 - 2 IntegerDigits[i - 1, 2, n^m] is shorter. – J. M. will be back soon Oct 27 '17 at 13:49
• @J.M. and much faster in the long run, thanks. – LLlAMnYP Oct 27 '17 at 13:51
• Even with {0,1} instead of {1,-1}, I haven't enough RAM to hold it. The same for PadLeft if the objective is to store all $2^{26}$ elements – Filburt Oct 27 '17 at 13:59
• @Filburt the point is that you don't have to hold it. If you know in advance how to generate any given element, that should be enough in most cases. I believe, your question is an example of an XY problem. Perhaps you could show us, what you want to do with your two sets? – LLlAMnYP Oct 27 '17 at 14:03
• @LLlAMnYP The other set is a little bit complicated to obtain. Is a set of $27\times 27$ matrices, and I want multiply each element of the second set with the transpose of each element from the bigset. You are right, I realize that we don't have to store bigset`. But maybe I will have no choice with the other one. The set is this: mathematica.stackexchange.com/questions/157585/… – Filburt Oct 27 '17 at 14:15