The problem here is that the exhibited Outer expression generates a 24-dimensional tensor that contains nearly 4 x 10^12 elements at the deepest level. There is no practical way to operate upon a structure of this size. Even simply iterating over the deepest elements will require a significant amount of time.
For discussion purposes, let us consider a smaller set of basis vectors when forming this outer product:
$small =
{ {q,-q}
, {p1, p2, p3}
, {4,5}
};
For this small example, it is feasible to evaluate the Outer expression:
Outer[func[##]&, Sequence @@ $small]
(*
{ { {func[q, p1, 4], func[q, p1, 5]}
, {func[q, p2, 4], func[q, p2, 5]}
, {func[q, p3, 4], func[q, p3, 5]}
}
, { {func[-q, p1, 4], func[-q, p1, 5]}
, {func[-q, p2, 4], func[-q, p2, 5]}
, {func[-q, p3, 4], func[-q, p3, 5]}
}
}
*)
The result is a 2 x 3 x 2 tensor. It is challenging to process such structures incrementally at anything other than the deepest level (the func[...] elements), especially for gigantic tensors. So we will focus upon generating the individual elements without regard to preserving the overall tensor structure.
We will define tupleSpigot, a facility for generating the individual tuples that comprise the deepest tensor level:
ClearAll[tupleSpigot]
tupleSpigot[l:{{__}..}] :=
With[{ns = Length/@l}, tupleSpigot[l, MixedRadix[ns], Times@@ns, Length@l]]
Length[tupleSpigot[_, _, n_, _]] ^:= n
tupleSpigot[l_, r_, n_, c_][[i_]] ^:=
MapThread[#[[#2]]&, {l, 1+IntegerDigits[i-1, r, c]}]
We can use this to represent all combinations of basis vectors from $small:
$s = tupleSpigot[$small]
(* tupleSpigot[{{q, -q}, {p1, p2, p3}, {4, 5}}, MixedRadix[{2, 3, 2}], 12, 3] *)
Note that this representation is relatively small. In particular it does not explicitly contain all of the generated tuples. But we can see that there are 12 elements as we saw from the earlier Outer expression:
Length[$s]
(* 12 *)
We can generate the tuples individually on demand, so that the whole structure is never materialized into memory all at once:
Do[Print[$s[[i]]], {i, 1, Length@$s}]
(*
{q,p1,4}
{q,p1,5}
{q,p2,4}
{q,p2,5}
{q,p3,4}
{q,p3,5}
{-q,p1,4}
{-q,p1,5}
{-q,p2,4}
{-q,p2,5}
{-q,p3,4}
{-q,p3,5}
*)
So, now let's try the same trick with the original full set of basis vectors from the question:
momentum[q_,p1_,p2_,p3_,p4_,p5_] :=
Flatten@Outer[{q,p1,p2,p3,p4,p5}.{##}&,{1,-1},{1,-1},{1,-1},{1,-1},{1,-1},{1,-1}]
$original =
{ {q,-q}
, {q}
, DeleteDuplicates[momentum[q,p1,p2,p3,p4,p5]]
, {p1}
, DeleteDuplicates[momentum[q,p1,p2,p3,p4,0]]
, {p2}
, DeleteDuplicates[momentum[q,p1,p2,p3,0,0]]
, {p3}
, DeleteDuplicates[momentum[q,p1,p2,0,0,0]]
, {p4}
, DeleteDuplicates[momentum[q,p1,0,0,0,0]]
, {p5}
, {0}
, {0}
, {0}
, {0}
, {0}
, {0}
, {-5,-4,-3,-2,-1,0,1,2,3,4,5}
, {-5,-4,-3,-2,-1,0,1,2,3,4,5}
, {-5,-4,-3,-2,-1,0,1,2,3,4,5}
, {-5,-4,-3,-2,-1,0,1,2,3,4,5}
, {-5,-4,-3,-2,-1,0,1,2,3,4,5}
, {-5,-4,-3,-2,-1,0,1,2,3,4,5}
};
$o = tupleSpigot[$original];
How many tuples does this generate?
Length@$o
(* 3715232694272 *)
Yikes, that is a big number. If each element were 1000 bytes long, then this would require 3715 terabytes of memory to store.
But we can access the elements individually. For brevity, we will only show the first and last three elements from this monstrous set:
Do[Print[$o[[i]]], {i, 1, 3}]
(*
{q,q,p1+p2+p3+p4+p5+q,p1,p1+p2+p3+p4+q,p2,p1+p2+p3+q,p3,p1+p2+q,p4,p1+q,p5,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5}
{q,q,p1+p2+p3+p4+p5+q,p1,p1+p2+p3+p4+q,p2,p1+p2+p3+q,p3,p1+p2+q,p4,p1+q,p5,0,0,0,0,0,0,-5,-5,-5,-5,-5,-4}
{q,q,p1+p2+p3+p4+p5+q,p1,p1+p2+p3+p4+q,p2,p1+p2+p3+q,p3,p1+p2+q,p4,p1+q,p5,0,0,0,0,0,0,-5,-5,-5,-5,-5,-3}
*)
Do[Print[$o[[i]]], {i, 3715232694270, 3715232694272}]
(*
{-q,q,-p1-p2-p3-p4-p5-q,p1,-p1-p2-p3-p4-q,p2,-p1-p2-p3-q,p3,-p1-p2-q,p4,-p1-q,p5,0,0,0,0,0,0,5,5,5,5,5,3}
{-q,q,-p1-p2-p3-p4-p5-q,p1,-p1-p2-p3-p4-q,p2,-p1-p2-p3-q,p3,-p1-p2-q,p4,-p1-q,p5,0,0,0,0,0,0,5,5,5,5,5,4}
{-q,q,-p1-p2-p3-p4-p5-q,p1,-p1-p2-p3-p4-q,p2,-p1-p2-p3-q,p3,-p1-p2-q,p4,-p1-q,p5,0,0,0,0,0,0,5,5,5,5,5,5}
*)
We might have overcome the memory problem, but if we want to process all those elements it is going to take a very long time. Somewhat optimistically, let's assume that we are running 256 parallel kernels and each kernel can process 10,000 elements per second. How many days will it take to perform the scan?
Length@$o / 256 / 10000 / 60 / 60 / 24 // N
(* 16.797 *)
If our expensive hardware doesn't die after pegging the CPUs at 100% for two straight weeks then maybe this is in the realm of possible. But it is certainly an expensive undertaking.
It would be fruitful look for an algorithmic solution to this problem. Will the domain allow us to reduce the dimensionality of the problem space so that we do not have to process so many elements? Can the we solve our problem, perhaps only probabilistically, by sampling a small subset of the large space?
