I aim to work with two sets, obtained through a function. One of these are:

n=3; m=3;
bigset=Take[Tuples[{1, -1}, n^m], 2^(n^m-1)];

(It calculates the set Tuples[{1, -1}, n^m] and takes the first half of it)

This set is easy to compute but hard to store (more than 100 GB, I think). I have only 8 GB of RAM. Every time I run this code, the computer freezes forever. Is there a way to calculate this big set using the hard disk space (taking no more than one night) and use it from there?

EDIT: The other set (called matrixset) can be obtained in the following way:


m = 3;
n = 3;

Inm = Tuples[Table[i, {i, 1, n}], m];
extBn = Tuples[{1, -1}, n];
prodBn = Tuples[Table[extBn, {m}]];
w[i_, j_] := Product[prodBn[[i]][[k]][[Inm[[j]][[k]]]], {k, 1, m}];
Vnm = DeleteDuplicates[
   Table[Table[w[i, j], {j, 1, Length[Inm]}], {i, 1, Length[prodBn]}]];
remove = Union[Tuples[{1}, n^m], Tuples[{-1}, n^m]];
V0 = {{Vnm[[1]]}};
S = Complement[Vnm, remove]; "";
S0 = Table[{S[[i]]}, {i, 1, Length[S]}]; ""; Clear[S];
f[S0_, {i_, V0_}] := Module[{S1, S2}, S1 = Join @@@ Tuples[{S0, V0}];
  S2 = Pick[DeleteDuplicatesBy[Sort]@S1, 
    MatrixRank[#] == i & /@ DeleteDuplicatesBy[Sort]@S1];
  {i + 1, S2}]
matrixset = Last@Nest[f[S0, #] &, {2, V0}, n^m - 1]; ""; Clear[S0];

The set consist of all invertible matrices $27\times 27$ that have the unit vector $(1,...,1)$ as the last line and vectors from the set $Vnm$ as the other lines. The process remove those that differ by row-switching.

The set S0 consists of 1-row matrices from $Vnm\backslash \{(1~\cdots~1), (-1~\cdots~-1)\}$. Here is how the Module work: (1) Prepend all vectors from S0 to the 1-line matrix $(1~\cdots~1)$; (2) obtain S1 by eliminating the matrices duplicated by row-switching and those with rank $< 2$; (3) Prepend all vectors from S0 to all matrices from S1; (4) obtain S2 by eliminating the matrices duplicated by row-switching and those with rank $< 3$; (5) and so on.....


1 Answer 1


Observation: If you replace Tuples[{1, -1}... with Tuples[{0, 1}... you'll be generating the base-2 integer digits of the numbers 0 through 2^(n^m-1)-1.

Therefore instead of generating the entire bigset you can simply realize that

bigset[[i]] == PadLeft[IntegerDigits[i - 1, 2], n^m] /. {0 -> 1, 1 -> -1}

or as J.M. points out, a much terser and performance-wise better variant of the expression above is

1 - 2 IntegerDigits[i - 1, 2, n^m]

Observations regarding memory consumption: you are in intermediate calculations generating a 2^27 * 27 array of integers (each of which are 64 bits in size). In principle, you could manage with 32 GB RAM or maybe slightly more. But since every 27-integer tuple is 216 bytes in size, when it could be represented by a single integer you are wasting memory by a factor of 27! Now my machine has also 8GB memory and generating range = Range[0, 2^26 - 1]; is no problem whatsoever and consumes only 500-something MB of memory.

  • $\begingroup$ 1 - 2 IntegerDigits[i - 1, 2, n^m] is shorter. $\endgroup$ Commented Oct 27, 2017 at 13:49
  • $\begingroup$ @J.M. and much faster in the long run, thanks. $\endgroup$
    – LLlAMnYP
    Commented Oct 27, 2017 at 13:51
  • $\begingroup$ Even with {0,1} instead of {1,-1}, I haven't enough RAM to hold it. The same for PadLeft if the objective is to store all $2^{26}$ elements $\endgroup$
    – Filburt
    Commented Oct 27, 2017 at 13:59
  • 2
    $\begingroup$ @Filburt the point is that you don't have to hold it. If you know in advance how to generate any given element, that should be enough in most cases. I believe, your question is an example of an XY problem. Perhaps you could show us, what you want to do with your two sets? $\endgroup$
    – LLlAMnYP
    Commented Oct 27, 2017 at 14:03
  • $\begingroup$ @LLlAMnYP The other set is a little bit complicated to obtain. Is a set of $27\times 27$ matrices, and I want multiply each element of the second set with the transpose of each element from the bigset. You are right, I realize that we don't have to store bigset. But maybe I will have no choice with the other one. The set is this: mathematica.stackexchange.com/questions/157585/… $\endgroup$
    – Filburt
    Commented Oct 27, 2017 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.