Skip to main content
1 of 4
Mr.Wizard
  • 273.1k
  • 34
  • 595
  • 1.4k

Since you are looking for duplicates you could adapt any of the methods shown in this answer.

Using the first one for example:

dupeQ =
  Catch[
    Module[{f},
      f[y_] := (f[y] := Throw[True]; y);
      Scan[f, #]
    ]; False
  ] &;

This particular one has an advantage on long lists in that it will "short-circuit" on the first duplicate found rather than proceeding through the end of the list as Union or DeleteDuplicates will. Example:

a = Range[1*^7];
a[[{1, 50}]] = 3.14159;

dupeQ[a] // Timing
{0., True}
a != DeleteDuplicates[a] // Timing
{5.944, True}

For this example I stacked the deck in my favor; by replacing elements in an Integer list with Reals I force the list to unpack. In this situation and with an early duplicate (positions 1 and 50) my method is about 65,000 times faster. For a case with a packed array dupeQ is slower than DeleteDuplicates but not as dramatically:

a = Range[1*^7];
a[[50]] = 7;

dupeQ[a] // Timing
{0.359, True}
a != DeleteDuplicates[a] // Timing
{0.031, True}
Mr.Wizard
  • 273.1k
  • 34
  • 595
  • 1.4k