1
$\begingroup$

I simplified my problem as follows, 

 nn = 2;
 ff[i_, j_, k_] := {i + j - k + cont (i - k), i k (i - j) g,Abs[i - j]/t, k q};
 lst0 = Flatten[Table[ff[i, j, k], {i, 1, nn}, {j, 1, nn}, {k, 1, nn}],2];

Now, I want to combine some sublists in lst0 and make the output to the text file. The rule is, if some sublists have the same elements except the first ones, they will be combined and their first elements should be added, other elements remain unchanged. For example, we can find these sublists using 

 GatherBy[lst0, #[[2 ;; 4]]&]

Then, we combine the sublists using some kind of function and get the final output to the file, 

{{4 + cont, 0, 0, q}, {2 - cont, 0, 0, 2 q}, {2, -g, 1/t, q}, {1 - cont, -2 g, 1/t, 2 q}, {2 + cont, 2 g, 1/t, q}, {1, 4 g, 1/t, 2 q}}

In the practical case, the function ff can produce very complicated expressions and nn, the dimension of list0 are also very large. So, the computation speed should be considered and also sometimes the kernel quits due to lack of memory. I am thinking we need to firstly output the list0 one by one, then read and transform it but I am not sure. So, considering the above, anyone can help ?

|improve this question
$\endgroup$
  • $\begingroup$ Will this work on your actual data? f[v_] := Join[{First[Plus @@ v]}, Rest[v[[1]]]];Map[f, GatherBy[lst0, #[[2 ;; 4]] &]] $\endgroup$ – Bill Aug 12 '16 at 4:22
  • $\begingroup$ @Bill Nice. Now, the problem is, if ff[] can produce a very large list, even out of memory, how do we deal with it ? $\endgroup$ – Orders Aug 12 '16 at 7:00
  • $\begingroup$ It is impossible to answer when you carefully avoid telling anyone what your large list is or how big it is. Given that, my first guess is: get 64 gigabytes of memory and see if that is enough. If you find that is not enough then produce parts of your table, perhaps 48 gigabytes worth at a time, use the method on each part and then write the result to your file. That should be much faster than doing this a single item at a time. $\endgroup$ – Bill Aug 12 '16 at 16:47
2
$\begingroup$

I would write this operation using Sow and Reap in a Do loop.

nn = 2;

ff[i_, j_, k_] :=
  Sow[
    i + j - k + cont (i - k),
    {{i k (i - j) g, Abs[i - j]/t, k q}}
  ]

Reap[
  Do[ff[i, j, k], {i, nn}, {j, nn}, {k, nn}],
  _,
  Prepend[#, Total @ #2] &
][[2]]

{
 {4 + cont, 0, 0, q},
 {2 - cont, 0, 0, 2 q},
 {2, -g, 1/t, q},
 {1 - cont, -2 g, 1/t, 2 q},
 {2 + cont, 2 g, 1/t, q},
 {1, 4 g, 1/t, 2 q}
}
|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.