Since you are looking for duplicates you could adapt any of the methods shown in this answer.
Using the first one for example:
dupeQ =
Catch[
Module[{f},
f[y_] := (f[y] := Throw[True]; y);
Scan[f, #]
]; False
] &;
This particular one has an advantage on long lists in that it will "short-circuit" on the first duplicate found rather than proceeding through the end of the list as Union
or DeleteDuplicates
will. Example:
a = Range[1*^7];
a[[{1, 50}]] = 3.14159;
dupeQ[a] // Timing
{0., True}
a != DeleteDuplicates[a] // Timing
{5.944, True}
For this example I stacked the deck in my favor; by replacing elements in an Integer list with Reals I force the list to unpack. In this situation and with an early duplicate (positions 1 and 50) my method is about 65,000 times faster. For a case with a packed array dupeQ
is slower than DeleteDuplicates
but not as dramatically:
a = Range[1*^7];
a[[50]] = 7;
dupeQ[a] // Timing
{0.359, True}
a != DeleteDuplicates[a] // Timing
{0.031, True}