How to merge some sublists in a very large list ?

Question

I simplified my problem as follows,

 nn = 2;
 ff[i_, j_, k_] := {i + j - k + cont (i - k), i k (i - j) g,Abs[i - j]/t, k q};
 lst0 = Flatten[Table[ff[i, j, k], {i, 1, nn}, {j, 1, nn}, {k, 1, nn}],2];

Now, I want to combine some sublists in lst0 and make the output to the text file. The rule is, if some sublists have the same elements except the first ones, they will be combined and their first elements should be added, other elements remain unchanged. For example, we can find these sublists using

 GatherBy[lst0, #[[2 ;; 4]]&]  

Then, we combine the sublists using some kind of function and get the final output to the file,

{{4 + cont, 0, 0, q}, {2 - cont, 0, 0, 2 q}, {2, -g, 1/t, q}, {1 - cont, -2 g, 1/t, 2 q}, {2 + cont, 2 g, 1/t, q}, {1, 4 g, 1/t, 2 q}}

In the practical case, the function ff can produce very complicated expressions and nn, the dimension of list0 are also very large. So, the computation speed should be considered and also sometimes the kernel quits due to lack of memory. I am thinking we need to firstly output the list0 one by one, then read and transform it but I am not sure. So, considering the above, anyone can help ?

Answer 1

I would write this operation using Sow and Reap in a Do loop.

nn = 2;

ff[i_, j_, k_] :=
  Sow[
    i + j - k + cont (i - k),
    {{i k (i - j) g, Abs[i - j]/t, k q}}
  ]

Reap[
  Do[ff[i, j, k], {i, nn}, {j, nn}, {k, nn}],
  _,
  Prepend[#, Total @ #2] &
][[2]]

{
 {4 + cont, 0, 0, q},
 {2 - cont, 0, 0, 2 q},
 {2, -g, 1/t, q},
 {1 - cont, -2 g, 1/t, 2 q},
 {2 + cont, 2 g, 1/t, q},
 {1, 4 g, 1/t, 2 q}
}