Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a list in this style

data={{a1,b1,c1,d1,e1,f1}, {a2,b2,c2,d2,e2,f2}}

I need to delete all elements in which b1 is equal to b2 and so on. I only have to test b. The list I use have more than 15000 elements. I used DeleteDuplicates:

DeleteDuplicates[data, {_, #1, _, _, _, _} == {_, #2, _, _, _, _} &]

I know this method gives a correct result because I tested it on a small amount of data.

But in a list with more than 15,000 elements after 20 minutes of running without success. Is there a more efficient method?

share|improve this question

marked as duplicate by bobthechemist, RunnyKine, Michael E2, Leonid Shifrin, Öskå Jul 31 at 21:44

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Have you searched the forum? There are some threads about his, for instance here mathematica.stackexchange.com/a/35214/187 –  halirutan Jul 31 at 19:43
@halirutan Nope, I didn't ;-D. Was too excited that I knew a v10 function. –  bobthechemist Jul 31 at 19:57
Haha...I am realy not againt leraning so can you explain me one of these faster methodes in the thread you mentioned? Like I am a real beginner....:) –  Antoine Mayer Jul 31 at 20:01

2 Answers 2

In version 10 we have DeleteDuplicatesBy which seems to be much faster:

t = RandomInteger[{1, 10}, {10^3, 5}];
   {_, #1, _, _, _, _} == {_, #2, _, _, _, _} &]; // AbsoluteTiming
DeleteDuplicatesBy[t, Part[#, 2] &] // AbsoluteTiming
(* 0.488, 0.001 *)
share|improve this answer
FYI, you mention c1/c2 in the text but appear to use b1/b2 in your example. –  bobthechemist Jul 31 at 19:44
Thanks I changed my example –  Antoine Mayer Jul 31 at 19:50
Okay I still use mathematica 9...I have to wait that school s begin again so I can get the 10....@ halirutan I saw other threads but I already have problems understanding what the & at the end is for so undertanding other exemples without explications is a bit hard :) :) –  Antoine Mayer Jul 31 at 19:56

Similar to Mr.Wizard's answer here**, we can use GatherBy

GatherBy[data, #[[2]] &][[All, 1]]

I must say I was surprised* how much faster than DeleteDuplicatesBy this is:

data = RandomInteger[1000, {15000, 5}];
DeleteDuplicatesBy[data, Part[#, 2] &]; // AbsoluteTiming
GatherBy[data, #[[2]] &][[All, 1]]; // AbsoluteTiming
  {12.071266, Null}
  {0.004069, Null}

With fewer duplicates:

data = RandomInteger[10000, {15000, 5}];
DeleteDuplicatesBy[data, Part[#, 2] &]; // AbsoluteTiming
GatherBy[data, #[[2]] &][[All, 1]]; // AbsoluteTiming
  {62.941921, Null}
  {0.017356, Null}

*That's because I forgot about this: DeleteDuplicatesBy is not performing as I'd hoped. Am I missing something?.

**And it turns out Mr.Wizard's answer to the question in the previous note * is also almost the same as this one.

share|improve this answer
+1. You beat me to it :) –  RunnyKine Jul 31 at 20:12
@RunnyKine It figures such a solution is already on the site. (And thanks!) –  Michael E2 Jul 31 at 20:22
Thank you tested the method of @MichaelE2 and it works like a charm...can you explain the difference between the two? –  Antoine Mayer Aug 1 at 12:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.