Focusing on the elimination of duplicates (vs. the generation of uniqueTuples
), I have two common functions I use in my notebooks, which when used together without making any assumptions about the input (ie. pairwise sublists or the input even being a list), provide a simple solution.
(* Eliminate duplicates of any permutation in a list *)
noDups@in_:=in;
noDups@in_List:=DeleteDuplicates[DeleteDuplicates@in,
If[#2=={},True,MemberQ[Permutations@#2,#1]]&];
(* Eliminate Nulls or empty lists or list elements with duplicate entries *)
noNull@in_List:=Select[DeleteCases[in,Null|{},2], Length@First@in==Length@noDups@#&];
Unfortunately, the //Timing
is about half as fast as Spawn1701D's general Union
based function. So if we make the assumption that the input can't be a single list of atomic elements (or Nulls or empty lists), we can use the following (the GatherBy
given by Mr.Wizard here) which has excellent //Timing
comparable even to the fastest solutions:
noDups@in_List := GatherBy[in~DeleteCases~{}, Sort][[All, 1]];
noNull@in_List := Select[in, Length@First@in == Length@Union@# &];
data = RandomInteger[80, {3, 40}];
t = Tuples@data;
noDups@noNull@t //Timing
{0.593750, {{69, 26, 32}, {69, 26, 59},...
Comparing the //Timing
for reducing (vs. generating unique) Tuples
on t
, we get 9.5 Seconds on this reducing algorithm (vs. 5.7 Seconds for Simon Woods generating algorithm) on:
data = RandomInteger[40, {6, 11}];
See the following for explicit comparison (**Note: the timing being the same between the two functions on this significantly smaller data set).
In[1]:= noDups@in_List := GatherBy[in~DeleteCases~{}, Sort][[All, 1]];
noNull@in_List := Select[in, Length@First@in == Length@Union@# &];
In[3]:= uniqueTuples[a_List, b_List] :=
Module[{f}, f[___, x_, x_, ___] = Sequence[];
f[x_, y__] := (f[x, y] = Sequence[]; {x, y});
SetAttributes[f, Orderless];
Flatten[Outer[f @@ Flatten[{##}] &, a, b, 1], 1]]
uniqueTuples[{a_List, x__List}] := Fold[uniqueTuples, a, {x}]
In[5]:= a = {1, 2, 3, 4, 6};
b = {2, 3, 4, 6, 9};
t = Tuples[{a, b}];
In[8]:= r1 = noDups@noNull@t
Out[8]= {{1, 2}, {1, 3}, {1, 4}, {1, 6}, {1, 9}, {2, 3}, {2, 4}, {2, 6}, {2, 9}, {3, 4}, {3, 6}, {3, 9}, {4, 6}, {4, 9}, {6, 9}}
In[9]:= r2 = uniqueTuples[{a, b}]
Out[9]= {{1, 2}, {1, 3}, {1, 4}, {1, 6}, {1, 9}, {2, 3}, {2, 4}, {2, 6}, {2, 9}, {3, 4}, {3, 6}, {3, 9}, {4, 6}, {4, 9}, {6, 9}}
In[10]:= r1 == r2
Out[10]= True
In[11]:= data = RandomInteger[40, {5, 11}];
In[12]:= t = Tuples@data;
In[13]:= r1 = noDups@noNull@t // Timing
Out[13]= {1.578125,{{34,38,20,0,23},{34,38,20,0,8}{34,38,20,0,2},{34,38,20,0,7},{34,38,20,0,25},{34,38,20,0,35},{34,38,20,0,14},<<54534>>,{5,10,38,11,2},{5,10,38,11,7},{5,10,38,11,25},{5,10,38,11,35},{5,10,38,11,14},{5,10,38,11,28},{5,10,38,11,17}}}
In[14]:= r2 = uniqueTuples@data // Timing
Out[14]= {1.578125,{{0,20,23,34,38},{0,8,20,34,38},{0,2,20,34,38},{0,7,20,34,38},{0,20,25,34,38},{0,20,34,35,38},{0,14,20,34,38},<<54534>>,{2,5,10,11,38},{5,7,10,11,38},{5,10,11,25,38},{5,10,11,35,38},{5,10,11,14,38},{5,10,11,28,38},{5,10,11,17,38}}}
In[15]:= Sort[Sort@# & /@ r1[[2]]] == Sort[Sort@# & /@ r2[[2]]]
Out[15]= True