Delete duplicate elements from a list

Question

If a list contains duplicate elements, for example

 list = {a, 1, 5, 3, 5, x^2, x^2},

how can the duplicate elements be removed? The result would be

 uniqueElements = {a, 1, 5, 3, x^2}

Answer 1

You can use DeleteDuplicates to remove the duplicate elements while preserving the original order:

DeleteDuplicates[{a, 1, 5, 3, 5, x^2, x^2}]
(* {a, 1, 5, 3, x^2} *)

Answer 2

In version 7 or later, use the DeleteDuplicates function. (See also DeleteDuplicatesBy, introduced in version 10, but be aware of performance considerations.)

For versions of Mathematica before 7, when DeleteDuplicates was introduced, and for general interest, here are several ways of implementing the UnsortedUnion (i.e. DeleteDuplicates) function. These are collected from the help docs and MathGroup. They have been adjusted to accept multiple lists which are then joined, in analogy to Union. Unlike Union, these functions do not sort the list in the process of removing duplicates.

These methods may be obsolete for the specific function of DeleteDuplicates but they demonstrate methods that continue to be useful in more general problems.

• Derivatives of the first method below using Sequence[]:

  1. DeleteDuplicates while retaining sublist structure
  2. Selecting minimal subsets

• The Sow/Reap method demonstrates sowing one object to multiple tags, the reverse of its most common use, to powerful effect.

• The Tally method can be generalized to GatherBy.

For Mathematica 4 or earlier [ref]

UnsortedUnion = Module[{f}, f[y_] := (f[y] = Sequence[]; y); f /@ Join@##] &

---

For Mathematica 5 [ref]

UnsortedUnion[x__List] := Reap[Sow[1, Join@x], _, # &][[2]]

---

For Mathematica 6

UnsortedUnion[x__List] := Tally[Join@x][[All, 1]]

---

From Leonid Shifrin for Mathematica 3+

unsortedUnion[x_List] := 
  Extract[x, Sort[Union[x] /. Dispatch[MapIndexed[Rule, x]]]]

Answer 3

If you don't care about the original order (or if you want it sorted), use Union:

Union@{a, 1, 5, 3, 5, x^2, x^2}
(* {1, 3, 5, a, x^2} *)

Answer 4

I use v5.2 regularly. The fastest way I've found to simulate DeleteDuplicates is

DD[data_] := Part[data,Sort@Part[Range[Length@data][[#]],
   Most@FoldList[Plus,1,Length/@Split@data[[#]]]]]& @ Ordering@data

Here are some comparative times. RS is from the Reap Help, UU is Carl Woll's UnsortedUnion, and VV is Leonid Shifrin's suggestion for v3+; all three have been mentioned earlier. WW, XX, and YY are my attempts to speed up VV.

RS[data_] := Reap[Sow[1, data], _, # &][[2]]

VV[data_] := Extract[data, Sort[Union[data] /. Dispatch[MapIndexed[Rule,data]]]]

WW[data_] := Part[data, Sort[Union[data] /. Dispatch[MapIndexed[#1->#2[[1]]&,data]]]]

XX[data_] := Part[data, Sort[Union[data] /. Dispatch[Thread@Rule[data,Range@Length@data]]]]

YY[data_] := Part[data, Sort[Union[data] /. Dispatch[MapThread[Rule,{data,Range@Length@data}]]]]

UU[data_] := Block[{f}, f[x_] := (f[x] = Sequence[]; x); f /@ data]

Length[data = Table[Random[Integer,1*^5],{1*^5}]]
Timing[rs = RS@data; "RS"]
Timing[vv = VV@data; "VV"]
Timing[ww = WW@data; "WW"]
Timing[xx = XX@data; "XX"]
Timing[yy = YY@data; "YY"]
Timing[uu = UU@data; "UU"]
Timing[dd = DD@data; "DD"]
{Length@dd, SameQ[dd,rs,uu,vv,ww,xx,yy]}

100000

{3.89 Second, RS}

{2.43 Second, VV}

{1.79 Second, WW}

{1.59 Second, XX}

{1.58 Second, YY}

{1.54 Second, UU}

{0.47 Second, DD}

{63070, True}