# How to merge some sublists in a very large list ?

I simplified my problem as follows,

 nn = 2;
ff[i_, j_, k_] := {i + j - k + cont (i - k), i k (i - j) g,Abs[i - j]/t, k q};
lst0 = Flatten[Table[ff[i, j, k], {i, 1, nn}, {j, 1, nn}, {k, 1, nn}],2];


Now, I want to combine some sublists in lst0 and make the output to the text file. The rule is, if some sublists have the same elements except the first ones, they will be combined and their first elements should be added, other elements remain unchanged. For example, we can find these sublists using

 GatherBy[lst0, #[[2 ;; 4]]&]


Then, we combine the sublists using some kind of function and get the final output to the file,

{{4 + cont, 0, 0, q}, {2 - cont, 0, 0, 2 q}, {2, -g, 1/t, q}, {1 - cont, -2 g, 1/t, 2 q}, {2 + cont, 2 g, 1/t, q}, {1, 4 g, 1/t, 2 q}}


In the practical case, the function ff can produce very complicated expressions and nn, the dimension of list0 are also very large. So, the computation speed should be considered and also sometimes the kernel quits due to lack of memory. I am thinking we need to firstly output the list0 one by one, then read and transform it but I am not sure. So, considering the above, anyone can help ?

• Will this work on your actual data? f[v_] := Join[{First[Plus @@ v]}, Rest[v[[1]]]];Map[f, GatherBy[lst0, #[[2 ;; 4]] &]] – Bill Aug 12 '16 at 4:22
• @Bill Nice. Now, the problem is, if ff[] can produce a very large list, even out of memory, how do we deal with it ? – Orders Aug 12 '16 at 7:00
• It is impossible to answer when you carefully avoid telling anyone what your large list is or how big it is. Given that, my first guess is: get 64 gigabytes of memory and see if that is enough. If you find that is not enough then produce parts of your table, perhaps 48 gigabytes worth at a time, use the method on each part and then write the result to your file. That should be much faster than doing this a single item at a time. – Bill Aug 12 '16 at 16:47

I would write this operation using Sow and Reap in a Do loop.

nn = 2;

ff[i_, j_, k_] :=
Sow[
i + j - k + cont (i - k),
{{i k (i - j) g, Abs[i - j]/t, k q}}
]

Reap[
Do[ff[i, j, k], {i, nn}, {j, nn}, {k, nn}],
_,
Prepend[#, Total @ #2] &
][[2]]

{
{4 + cont, 0, 0, q},
{2 - cont, 0, 0, 2 q},
{2, -g, 1/t, q},
{1 - cont, -2 g, 1/t, 2 q},
{2 + cont, 2 g, 1/t, q},
{1, 4 g, 1/t, 2 q}
}