# How to remove duplicates from a pair with some conditions?

Here is a simpler example of my real problem. Assume that I have a groups which includes sublists as follows:

groups = {{{a, b, c}, {d, e, f}}, {{b, a, c}, {g, h, k }}, {{e, d,
f}, {a, k, h}}};
(*groups = {group1, group2, group3}*)


The size of groups and each sublist groupi aren't necessary to be 3 as in the example.

list = Flatten[groups, 1]
(*list= {{a,b,c},{d,e,f},{b,a,c},{g,h,k},{e,d,f},{a,k,h}}*)


Now in the list above I consider pairs such as {a,b,c} and {b,a,c}, {d,e,f} and {e,d,f} as duplicates and I want to remove one from each pair. However, the removal is based on some conditions rather than just randomly remove one of them in each pair. The condition here is that I want to maximize removing of more elements from a group if possible. So for example with these two pairs,

(*
pair1= {a,b,c}, {b,a,c};
pair2= {d,e,f}, {e,d,f}
*)


I want to remove {a,b,c} instead of {b,a,c} from pair1 and {d,e,f} from pair2 instead of {e,d,f} as this remove completely group1 above this helps to maximize the removal of elements from one group. The expected output would be this:

expected_out = {{b, a, c}, {g, h, k}, {e, d, f}, {a, k, h}}


However, if I just use DeleteDuplicates I got this which is not something I'm looking for.

DeleteDuplicates[list, #1[[1 ;; 2]] == Reverse[#2[[1 ;; 2]]] &]
(*{{a,b,c},{d,e,f},{g,h,k},{a,k,h}}*)

• If I understand it correctly, you could as easily choose group2 to be decimated in your example by deleting {b, a, c} and {g, h, k} .How is the target group specified/chosen?
– Syed
May 8, 2022 at 19:22
• @Syed {g, h, k} is not removed as there is no {h,g,k}. So you could delete {b, a, c} but this make it that you deleted one element from group1 and one element from group2 instead of two elements from group1. May 8, 2022 at 19:24
• I want to maximize the number of elements removed from one group so I want to remove elemets from group1 instead. May 8, 2022 at 19:34
• Does this work on a larger dataset? res = Last /@ GatherBy[list, Union].
– Syed
May 8, 2022 at 19:37
• @Syed no, as the order could be different and data could be numbers as well. May 8, 2022 at 19:38

Here's a stab, but I suspect it can be made more elegant. I'll walk through the development step by step before putting together a complete function.

groups = {{{a, b, c}, {d, e, f}}, {{b, a, c}, {g, h, k}}, {{e, d, f}, {a, k, h}}};
elems = Union[Catenate[groups]];(* not sure if Union is necessary in your context *)


We need to identify dups. Given the main structure and a particular element, we can count the number of times a permutation of that element appears. We codify that in a function.

MatchingPositions[struct_][elem_] :=
With[
{normedStruct = Map[Sort, struct, {-2}],
normedElem = Sort[elem]},
Position[normedStruct, normedElem]]
(* comment: I had originally thought I'd use these positions to perform the later deletions, but as it turned out, that was unnecessary, and this implementation is more complicated than it needs to be. *)


And now:

dupElems = Select[elems, 1 < Length[MatchingPositions[groups][#]] &]
(* {{a, b, c}, {b, a, c}, {d, e, f}, {e, d, f}} *)


For each duplicate group, we want to select one as the candidate for deletion, and then we want to find all combinations of such candidates across all duplicate groups.

candidates = Tuples[GatherBy[dupElems, Sort]]
(* {{{a, b, c}, {d, e, f}}, {{a, b, c}, {e, d, f}}, {{b, a, c}, {d, e, f}}, {{b, a, c}, {e, d, f}}} *)


At this point, I'm not entirely sure whether I've correctly interpreted your preference criteria, and I'm also not convinced that it's well defined for all possible groups, but maybe there are unstated constraints in your actual problem setup. Anyway, I've decided that we want to minimize what I call "group span".

GroupSpan[struct_][elems_] :=
Length[CountsBy[Catenate[Position[struct, #] & /@ elems], First]];
(* testing it out... *)
GroupSpan[groups] /@ candidates
(* {1, 2, 2, 2} *)


We can now select the optimal dup-deletion strategy (might be more than one):

MinimalBy[candidates, GroupSpan[groups]]
(* {{{a, b, c}, {d, e, f}}} *)


Putting it all together, we can define a function that suggests candidates for deletion:

DuplicatesToDelete[struct_] :=
With[
{elemsx = Union[Catenate[struct]]},
With[
{dupElemsx = Select[elemsx, 1 < Length[MatchingPositions[struct][#]] &]},
With[
{candidatesx = Tuples[GatherBy[dupElemsx, Sort]]},
MinimalBy[candidatesx, GroupSpan[struct]]]]]


We can apply this to do the deletion, selecting the first (in this case only) candidate.

DeleteCases[groups, Alternatives @@ First[DuplicatesToDelete[groups]], 2]
(* {{}, {{b, a, c}, {g, h, k}}, {{e, d, f}, {a, k, h}}} *)


I left the empty sub-group for explicitness, but of course that could be removed to get your expected output.

How about listNoDuplicates = DeleteDuplicates[Sort /@ list]? Each list inside list gets sorted with Sort/@ and then the duplicates get deleted.