If a list contains duplicate elements, for example

 list = {a, 1, 5, 3, 5, x^2, x^2},

how can the duplicate elements be removed? The result would be

 uniqueElements = {a, 1, 5, 3, x^2}

4 Answers 4


You can use DeleteDuplicates to remove the duplicate elements while preserving the original order:

DeleteDuplicates[{a, 1, 5, 3, 5, x^2, x^2}]
(* {a, 1, 5, 3, x^2} *)

In version 7 or later, use the DeleteDuplicates function. (See also DeleteDuplicatesBy, introduced in version 10, but be aware of performance considerations.)

For versions of Mathematica before 7, when DeleteDuplicates was introduced, and for general interest, here are several ways of implementing the UnsortedUnion (i.e. DeleteDuplicates) function. These are collected from the help docs and MathGroup. They have been adjusted to accept multiple lists which are then joined, in analogy to Union. Unlike Union, these functions do not sort the list in the process of removing duplicates.

These methods may be obsolete for the specific function of DeleteDuplicates but they demonstrate methods that continue to be useful in more general problems.

For Mathematica 4 or earlier [ref]

UnsortedUnion = Module[{f}, f[y_] := (f[y] = Sequence[]; y); f /@ Join@##] &

For Mathematica 5 [ref]

UnsortedUnion[x__List] := Reap[Sow[1, Join@x], _, # &][[2]]

For Mathematica 6

UnsortedUnion[x__List] := Tally[Join@x][[All, 1]]

From Leonid Shifrin for Mathematica 3+

unsortedUnion[x_List] := 
  Extract[x, Sort[Union[x] /. Dispatch[MapIndexed[Rule, x]]]]
  • 1
    $\begingroup$ Wizard I like the Leonid's solution for the old versions of Mathematica. Not too relevant anymore, but still cool. $\endgroup$
    – Pillsy
    Commented Feb 28, 2012 at 16:33
  • 2
    $\begingroup$ +1. Actually, I had an impression that Dispatch first appeared only in 4.1, but I may be mistaken. Also, IIRC, the first solution is a version of the solution first published by Carl Woll. $\endgroup$ Commented Feb 28, 2012 at 16:59
  • $\begingroup$ @LeonidShifrin The first two solutions are also found in the documentations of the respective versions. That is where I learned about them. The docs say Dispatch is from v2, but I start using Mma only at v4, and at that time I didn't even know about Dispatch, so I don't know if it changed. $\endgroup$
    – Szabolcs
    Commented Feb 28, 2012 at 17:18
  • 1
    $\begingroup$ @Szabolcs Interesting. I knew that the Reap-Sow one is in the docs, but not about the one by Carl. You may be right about Dispatch. $\endgroup$ Commented Feb 28, 2012 at 17:38
  • $\begingroup$ Being a fan of Reap and Sow, I like the v.5 code as it is most likely a single pass operation. Similarly, the v.6 code is likely to be single pass, also. $\endgroup$
    – rcollyer
    Commented Mar 1, 2012 at 15:41

If you don't care about the original order (or if you want it sorted), use Union:

Union@{a, 1, 5, 3, 5, x^2, x^2}
(* {1, 3, 5, a, x^2} *)
  • 8
    $\begingroup$ Since DeleteDuplicates is faster than Union I would say this is if you do care about sorting, specifically that you want it. $\endgroup$
    – Mr.Wizard
    Commented Feb 28, 2012 at 16:05
  • 3
    $\begingroup$ @Mr.Wizard: Fair enough, but then I should mention that with Union you have to type less :) $\endgroup$ Commented Feb 28, 2012 at 17:16
  • 7
    $\begingroup$ Haha, true, but if you don't feel like typing the best advice is to not use Mathematica... NotebookSetupLayoutInformationPacket, MultivariateHypergeometricDistribution, NormalizedSquaredEuclideanDistance or the ironic VerboseConvertToPostScriptPacket $\endgroup$
    – Rojo
    Commented Feb 28, 2012 at 18:02
  • $\begingroup$ @Rojo reminds me of generic programming in c++. $\endgroup$
    – rcollyer
    Commented Feb 29, 2012 at 17:32

I use v5.2 regularly. The fastest way I've found to simulate DeleteDuplicates is

DD[data_] := Part[data,Sort@Part[Range[Length@data][[#]],
   Most@FoldList[Plus,1,Length/@Split@data[[#]]]]]& @ Ordering@data

Here are some comparative times. RS is from the Reap Help, UU is Carl Woll's UnsortedUnion, and VV is Leonid Shifrin's suggestion for v3+; all three have been mentioned earlier. WW, XX, and YY are my attempts to speed up VV.

RS[data_] := Reap[Sow[1, data], _, # &][[2]]

VV[data_] := Extract[data, Sort[Union[data] /. Dispatch[MapIndexed[Rule,data]]]]

WW[data_] := Part[data, Sort[Union[data] /. Dispatch[MapIndexed[#1->#2[[1]]&,data]]]]

XX[data_] := Part[data, Sort[Union[data] /. Dispatch[Thread@Rule[data,Range@Length@data]]]]

YY[data_] := Part[data, Sort[Union[data] /. Dispatch[MapThread[Rule,{data,Range@Length@data}]]]]

UU[data_] := Block[{f}, f[x_] := (f[x] = Sequence[]; x); f /@ data]

Length[data = Table[Random[Integer,1*^5],{1*^5}]]
Timing[rs = RS@data; "RS"]
Timing[vv = VV@data; "VV"]
Timing[ww = WW@data; "WW"]
Timing[xx = XX@data; "XX"]
Timing[yy = YY@data; "YY"]
Timing[uu = UU@data; "UU"]
Timing[dd = DD@data; "DD"]
{Length@dd, SameQ[dd,rs,uu,vv,ww,xx,yy]}


{3.89 Second, RS}

{2.43 Second, VV}

{1.79 Second, WW}

{1.59 Second, XX}

{1.58 Second, YY}

{1.54 Second, UU}

{0.47 Second, DD}

{63070, True}

  • $\begingroup$ Very interesting. I don't think I've seen your method before, though it has common features with other code I've seen and used. I wonder if this can be improved further? $\endgroup$
    – Mr.Wizard
    Commented Nov 2, 2013 at 1:53
  • $\begingroup$ I expect it will be eventually, but it doesn’t seem to be a hot topic so it may take a while. $\endgroup$ Commented Nov 2, 2013 at 21:17
  • $\begingroup$ To give credit where it's due, your recent comment drew my attention to Leonid's answer, which I am pretty sure my DD above is just an adaptation of. $\endgroup$ Commented Dec 14, 2013 at 10:41
  • $\begingroup$ Range[Length@data][[#]] in DD can be simplified as #. $\endgroup$
    – luyuwuli
    Commented Nov 5, 2016 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.