I simplified my problem as follows,

 nn = 2;
 ff[i_, j_, k_] := {i + j - k + cont (i - k), i k (i - j) g,Abs[i - j]/t, k q};
 lst0 = Flatten[Table[ff[i, j, k], {i, 1, nn}, {j, 1, nn}, {k, 1, nn}],2];

Now, I want to combine some sublists in lst0 and make the output to the text file. The rule is, if some sublists have the same elements except the first ones, they will be combined and their first elements should be added, other elements remain unchanged. For example, we can find these sublists using

 GatherBy[lst0, #[[2 ;; 4]]&]  

Then, we combine the sublists using some kind of function and get the final output to the file,

{{4 + cont, 0, 0, q}, {2 - cont, 0, 0, 2 q}, {2, -g, 1/t, q}, {1 - cont, -2 g, 1/t, 2 q}, {2 + cont, 2 g, 1/t, q}, {1, 4 g, 1/t, 2 q}}

In the practical case, the function ff can produce very complicated expressions and nn, the dimension of list0 are also very large. So, the computation speed should be considered and also sometimes the kernel quits due to lack of memory. I am thinking we need to firstly output the list0 one by one, then read and transform it but I am not sure. So, considering the above, anyone can help ?

  • $\begingroup$ Will this work on your actual data? f[v_] := Join[{First[Plus @@ v]}, Rest[v[[1]]]];Map[f, GatherBy[lst0, #[[2 ;; 4]] &]] $\endgroup$
    – Bill
    Commented Aug 12, 2016 at 4:22
  • $\begingroup$ @Bill Nice. Now, the problem is, if ff[] can produce a very large list, even out of memory, how do we deal with it ? $\endgroup$
    – Orders
    Commented Aug 12, 2016 at 7:00
  • $\begingroup$ It is impossible to answer when you carefully avoid telling anyone what your large list is or how big it is. Given that, my first guess is: get 64 gigabytes of memory and see if that is enough. If you find that is not enough then produce parts of your table, perhaps 48 gigabytes worth at a time, use the method on each part and then write the result to your file. That should be much faster than doing this a single item at a time. $\endgroup$
    – Bill
    Commented Aug 12, 2016 at 16:47

1 Answer 1


I would write this operation using Sow and Reap in a Do loop.

nn = 2;

ff[i_, j_, k_] :=
    i + j - k + cont (i - k),
    {{i k (i - j) g, Abs[i - j]/t, k q}}

  Do[ff[i, j, k], {i, nn}, {j, nn}, {k, nn}],
  Prepend[#, Total @ #2] &
 {4 + cont, 0, 0, q},
 {2 - cont, 0, 0, 2 q},
 {2, -g, 1/t, q},
 {1 - cont, -2 g, 1/t, 2 q},
 {2 + cont, 2 g, 1/t, q},
 {1, 4 g, 1/t, 2 q}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.