# Faster Alternative to DeleteDuplicates for deleting elements in list [duplicate]

I have a list in this style

data={{a1,b1,c1,d1,e1,f1}, {a2,b2,c2,d2,e2,f2}}


I need to delete all elements in which b1 is equal to b2 and so on. I only have to test b. The list I use have more than 15000 elements. I used DeleteDuplicates:

DeleteDuplicates[data, {_, #1, _, _, _, _} == {_, #2, _, _, _, _} &]


I know this method gives a correct result because I tested it on a small amount of data.

But in a list with more than 15,000 elements after 20 minutes of running without success. Is there a more efficient method?

-

## marked as duplicate by bobthechemist, RunnyKine, Michael E2, Leonid Shifrin, ÖskåJul 31 '14 at 21:44

Have you searched the forum? There are some threads about his, for instance here mathematica.stackexchange.com/a/35214/187 –  halirutan Jul 31 '14 at 19:43
@halirutan Nope, I didn't ;-D. Was too excited that I knew a v10 function. –  bobthechemist Jul 31 '14 at 19:57
Haha...I am realy not againt leraning so can you explain me one of these faster methodes in the thread you mentioned? Like I am a real beginner....:) –  Antoine Mayer Jul 31 '14 at 20:01

In version 10 we have DeleteDuplicatesBy which seems to be much faster:

t = RandomInteger[{1, 10}, {10^3, 5}];
DeleteDuplicates[t,
{_, #1, _, _, _, _} == {_, #2, _, _, _, _} &]; // AbsoluteTiming
DeleteDuplicatesBy[t, Part[#, 2] &] // AbsoluteTiming
(* 0.488, 0.001 *)

-
FYI, you mention c1/c2 in the text but appear to use b1/b2 in your example. –  bobthechemist Jul 31 '14 at 19:44
Thanks I changed my example –  Antoine Mayer Jul 31 '14 at 19:50
Okay I still use mathematica 9...I have to wait that school s begin again so I can get the 10....@ halirutan I saw other threads but I already have problems understanding what the & at the end is for so undertanding other exemples without explications is a bit hard :) :) –  Antoine Mayer Jul 31 '14 at 19:56

Similar to Mr.Wizard's answer here**, we can use GatherBy

GatherBy[data, #[[2]] &][[All, 1]]


I must say I was surprised* how much faster than DeleteDuplicatesBy this is:

data = RandomInteger[1000, {15000, 5}];
DeleteDuplicatesBy[data, Part[#, 2] &]; // AbsoluteTiming
GatherBy[data, #[[2]] &][[All, 1]]; // AbsoluteTiming
(*
{12.071266, Null}
{0.004069, Null}
*)


With fewer duplicates:

data = RandomInteger[10000, {15000, 5}];
DeleteDuplicatesBy[data, Part[#, 2] &]; // AbsoluteTiming
GatherBy[data, #[[2]] &][[All, 1]]; // AbsoluteTiming
(*
{62.941921, Null}
{0.017356, Null}
*)


*That's because I forgot about this: DeleteDuplicatesBy is not performing as I'd hoped. Am I missing something?.

**And it turns out Mr.Wizard's answer to the question in the previous note * is also almost the same as this one.

-
+1. You beat me to it :) –  RunnyKine Jul 31 '14 at 20:12
@RunnyKine It figures such a solution is already on the site. (And thanks!) –  Michael E2 Jul 31 '14 at 20:22
Thank you tested the method of @MichaelE2 and it works like a charm...can you explain the difference between the two? –  Antoine Mayer Aug 1 '14 at 12:18