# Re-arranging / selecting of a data table very slow in For loop [duplicate]

My task is quite simple; I have a table of data like

M = {{1,2}, {1,3}, {3,4}, {5,1}, {6,9}, {6,8}, {6,1}};


so the table is sorted with respect to the first entry of each tuple. What I want is basically to gather tuples with the same 'x' entry in a new one. M would look like this:

M = {{1,(2+3)/2}, {3,4}, {5,1}, {6,(9+8+1)/3}};


This can be possibly solved elegantly with some gathering and application of summation and counting, but since I'm a C speaker, I've taken the For loop as my first try:

Stringed = ConstantArray[{-1., -1.}, Dimensions[StoreFlat][[1]]]; (* this is
the array where all the data will be stored in. StoreFlat is the sorted but
"uncollected" input *)
Module[{i = 1, j = 1, k = 1, a = 1., f = 1.},
Monitor[
For[i = 1, i <= Dimensions[StoreFlat][[1]], i++, (* loop over the number of tuples *)
If[i == 1, (* first entry *)
f = StoreFlat[[1, 1]];
a = StoreFlat[[1, 2]];,
If[f == StoreFlat[[i, 1]], (* a tuple exists already with this entry *)
a += StoreFlat[[i, 2]];
k++;,
f = StoreFlat[[i, 1]]; (* this entry is larger than the former one *)
a = StoreFlat[[i, 2]];
j++; k = 1;
];
];
Stringed[[j]] = {f, a/k}; (* output. the second entry is divided by the number of
entries. *)
],
ProgressIndicator@N[i/Dimensions[StoreFlat][[1]]]
]
];


I know that this is probably not the fastest way. But since my input is not too larger (190 000 tuples of a real, floating point value) this should do the job. Is anything wrong about my approach? On my laptop, this thing takes a lot of time (10 to 20 minutes, I guess - I cancelled it).

• Have a look at GatherBy. Your task can be implemented as a one-liner, and be fast. If you want to use the language effectively, you need to use its own idioms, rather than porting code from other languages you already know. – Leonid Shifrin Mar 2 '14 at 10:49
• Thanks Leonid, I used Stringed = GatherBy[StoreFlat, #[[1]] &]; Map[{#[[1, 1]], Mean[#[[All, -1]]]} &, Stringed] to solve my problem! – Clemens Mar 2 '14 at 11:14
• @Clemens Could you explain why should anyone answer your questions? You haven't upvoted any answer so far. If answers are not satisfactory you should explain why. Right? – Artes Mar 30 '14 at 22:02
• I've upvoted Leonid Shifrin's answer right at the 2nd March! GatherBy was all I needed and I posted my solution. Hope that was not offending someone. – Clemens Mar 31 '14 at 6:14

As L.S. said, using MMA like C is a path to poor performance.

Based on the OP content, I took the liberty of assuming you want the Mean of remaining tuple values appended to the first:

d = {{1, 2}, {1, 3}, {3, 4}, {5, 1}, {6, 9}, {6, 8}, {6, 1}};
Map[{#[[1, 1]], Mean[#[[All, 2]]]} &, GatherBy[d, First]]

test = RandomInteger[{1, 20}, {100000, 2}];
Map[{#[[1, 1]], Mean[#[[All, 2]]]} &, GatherBy[test, First]]; // Timing

(*
{{1,5/2},{3,4},{5,1},{6,6}}
{0.062400,Null}
*)


N.B.: that time of 6/100th of a second for 100000 tuples is on a netbook...

A quick explanation of the workings: Everything (well, practically everything) in Mathematica revolves around lists. The GatherBy[...,First] takes the list of your tuples, and gathers them based on the value of the first elements, giving you a list of lists, each having contents of tuples with the same first element.

Then Map is used to go over that list, and for each entry (a list of tuples with same firsts), it takes the first element of the first entry (the leading result element), and then takes the second element of any and all members of the tuple list and computes the Mean. The two results make up a list which is progressively built up to the final result.

I'd recommend highly looking at the thread of common pitfalls here, and once you've gotten comfortable with the language, Leonid Shifrin's guide to advanced programming here.

• very helpful answer +1 – ubpdqn Mar 2 '14 at 12:00
• It's slightly quicker, at least on my PC with this data, to just use Mean /@ GatherBy[test, First] – Simon Woods Mar 2 '14 at 12:24
• @SimonWoods: yes. I didn't write a super-terse form, figured list handling in baby-steps... +1 on comment/ – ciao Mar 2 '14 at 22:39
{ First @ First @ #, Mean @ Last @ Transpose @ #}& /@ GatherBy[ M, First]

{{1, 5/2}, {3, 4}, {5, 1}, {6, 6}}


or

{First @ #1, Mean @ #2}& @@@ Transpose /@ GatherBy[ M, First]


See (Apply (at the first level) - @@@, Map - /@, Slot - #)

An approach using Reap and Sow:

Last@Reap[Sow@@@(Reverse/@M),_, {#1,Mean@#2}&]

Thank you rasher for correcting my initial post (see comments)

• Need Last@ on the front... and one of the few cases where Sow/Reap is slower. Pretty though! – ciao Mar 2 '14 at 11:38
• @rasher yes...doing from iPhone so messy...sorry – ubpdqn Mar 2 '14 at 11:41