1
$\begingroup$

Some time ago I asked the following question:

Merge list repeating elements

I was easily answered and I was satisfied by the answer. However in computing such combinatorics, I saturated the RAM very easily, both in my computer and in a cluster.

I would like to know if there is a way to save the output coming from Distribute while Mathematica is running maybe in different files (one for component, I was thinking, but maybe there is also some more sophisticated way), so that it does not occupy the RAM and I can later call part of the output when I need it, and the RAM saturation should be solved.

I'm also open to the possibility that there're other ways to store the large output coming from Distribute (or equivalent) so that I can actually do the computation without problem.

Improve this question
$\endgroup$
13
  • 3
    $\begingroup$ I have the strong feeling that this is an X-Y question. By using Outer, Tuples, or Distribute from the other answer, you just bloat your memory without getting any further bit of information. Why don't you appreciate that the data was seemingly well compressed in the first place? My suffestions: Find a way to generate small chunks of the uncompressed data from the compressed one as needed. $\endgroup$
    – Henrik Schumacher
    Commented Feb 27, 2020 at 20:57
  • $\begingroup$ Also another memory-friendly suggestion: Try to avoid symbols in large arrays. Instead, index your symbols by integers (you can use a simple flat list as dictionary and its PositionIndex as dictionary). Then you have the chance to work with packed arrays which can be stored and processed very efficiently. Plus, you can employ Compile to speed up procedures on your data. $\endgroup$
    – Henrik Schumacher
    Commented Feb 27, 2020 at 21:04
  • $\begingroup$ Do you really need to generate all the tuples, to only use (part of) them later? Why not just generate the tuples you need by index? $\endgroup$
    – ciao
    Commented Feb 27, 2020 at 21:58
  • 1
    $\begingroup$ My idea was maybe to generate them and use them and then discard them maybe. So that I don't need to save them. It's an intermediate step in my code, so I just need the tuple to manipulate later in different functions, but I need to repeat such manipulation for any Tuple $\endgroup$
    – Alessandro Mininno
    Commented Feb 28, 2020 at 8:15
  • 1
    $\begingroup$ @AlessandroMininno - in that case, just write a simple function to generate the tuples, say by index, and call that function with the sequence of index values. That said, I have the same feeling as Henrik that this may be an X-Y question: What precisely is the end game here? Are you generating some combinatorial structure and then counting members that meet some criteria? If so, why not just do it directly using combinatorial means? $\endgroup$
    – ciao
    Commented Feb 28, 2020 at 8:21

1 Answer 1

Reset to default
2
$\begingroup$

Here's a quick-n-dirty way to generate the tuples by index. This will handle things that would be preposterous to try to generate first using Tuples. This should give you a start, it can be easily modified to generate tuples in batches of arbitrary size.

tups[l_, n_] := Module[{l1 = Length@l,l2 = Rest[Length /@ l]}, 
   Extract[l, Transpose[{Range@l1, IntegerDigits[n - 1, MixedRadix[l2], l1] + 1}]]];

An example tuples generation list of 20 elements, each of 10 elements, each of two elements:

example = ArrayReshape[Range@400, {20, 10, 2}];

This would generate 10^20 tuples...

First tuple:

tups[example, 1] // Short // AbsoluteTiming

{0.0002863,{{1,2},{21,22},{41,42},{61,62},<<12>>,{321,322},{341,342},{361,362},{381,382}}}

A random deep tuple:

tups[example, 95675769776785995775] // Short // AbsoluteTiming

{0.0002038,{{19,20},{31,32},{53,54},{75,76},{91,92},<<11>>,{331,332},{355,356},{375,376},{389,390}}}

Improve this answer
$\endgroup$
1
  • $\begingroup$ That's great! So I just need to know the number of tuples that I expect, and the function compute such combination in zero time. I think that this is exactly what I was looking for. I'll try it on some of my examples! Thank you! $\endgroup$
    – Alessandro Mininno
    Commented Feb 28, 2020 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.