# Manage to save large arrays

Some time ago I asked the following question:

Merge list repeating elements

I was easily answered and I was satisfied by the answer. However in computing such combinatorics, I saturated the RAM very easily, both in my computer and in a cluster.

I would like to know if there is a way to save the output coming from Distribute while Mathematica is running maybe in different files (one for component, I was thinking, but maybe there is also some more sophisticated way), so that it does not occupy the RAM and I can later call part of the output when I need it, and the RAM saturation should be solved.

I'm also open to the possibility that there're other ways to store the large output coming from Distribute (or equivalent) so that I can actually do the computation without problem.

• I have the strong feeling that this is an X-Y question. By using Outer, Tuples, or Distribute from the other answer, you just bloat your memory without getting any further bit of information. Why don't you appreciate that the data was seemingly well compressed in the first place? My suffestions: Find a way to generate small chunks of the uncompressed data from the compressed one as needed. – Henrik Schumacher Feb 27 '20 at 20:57
• Also another memory-friendly suggestion: Try to avoid symbols in large arrays. Instead, index your symbols by integers (you can use a simple flat list as dictionary and its PositionIndex as dictionary). Then you have the chance to work with packed arrays which can be stored and processed very efficiently. Plus, you can employ Compile to speed up procedures on your data. – Henrik Schumacher Feb 27 '20 at 21:04
• Do you really need to generate all the tuples, to only use (part of) them later? Why not just generate the tuples you need by index? – ciao Feb 27 '20 at 21:58
• @ciao, yes I need all the tuples. What I meant was that I need to call them one by one later, but I need to call them all. So I cannot generate only those I need because I need them all. – Alessandro Mininno Feb 28 '20 at 8:14
• @AlessandroMininno - in that case, just write a simple function to generate the tuples, say by index, and call that function with the sequence of index values. That said, I have the same feeling as Henrik that this may be an X-Y question: What precisely is the end game here? Are you generating some combinatorial structure and then counting members that meet some criteria? If so, why not just do it directly using combinatorial means? – ciao Feb 28 '20 at 8:21

Here's a quick-n-dirty way to generate the tuples by index. This will handle things that would be preposterous to try to generate first using Tuples. This should give you a start, it can be easily modified to generate tuples in batches of arbitrary size.

tups[l_, n_] := Module[{l1 = Length@l,l2 = Rest[Length /@ l]},
Extract[l, Transpose[{Range@l1, IntegerDigits[n - 1, MixedRadix[l2], l1] + 1}]]];


An example tuples generation list of 20 elements, each of 10 elements, each of two elements:

example = ArrayReshape[Range@400, {20, 10, 2}];


This would generate 10^20 tuples...

First tuple:

tups[example, 1] // Short // AbsoluteTiming


{0.0002863,{{1,2},{21,22},{41,42},{61,62},<<12>>,{321,322},{341,342},{361,362},{381,382}}}

A random deep tuple:

tups[example, 95675769776785995775] // Short // AbsoluteTiming


{0.0002038,{{19,20},{31,32},{53,54},{75,76},{91,92},<<11>>,{331,332},{355,356},{375,376},{389,390}}}

• That's great! So I just need to know the number of tuples that I expect, and the function compute such combination in zero time. I think that this is exactly what I was looking for. I'll try it on some of my examples! Thank you! – Alessandro Mininno Feb 28 '20 at 9:36