# Delete duplicate from list of lists with unequal lengths

I have a list of lists with different lengths, it is of the form

{
{{1,2},{3,4}},
{{5,6},{7,8},{25,-1},{-14,-3}},
{{1,2}},
{{5,6},{7,8},{22,-1},{-11,-3}},
}


It contains duplicates. What is the fastset way to delete all duplicates ?

I want :

{
{{1,2},{3,4}},
{{5,6},{7,8},{25,-1},{-14,-3}},
{},
{{22,-1},{-11,-3}},
}


or

{
{{3,4}},
{{25,-1},{-14,-3}},
{{1,2}},
{{5,6},{7,8},{22,-1},{-11,-3}},
}


Or whatever you want as long as there is only one representative for each duplicate.

The test list:

   {{{57, 112}, {58, 113}, {59, 113}, {60, 114}, {61, 114}, {62,
115}, {63, 116}}, {{58, 111}, {59, 112}, {60, 112}, {61, 113}, {62,
113}, {63, 114}, {64, 115}}, {{58, 110}, {59, 111}, {60,
111}, {61, 112}, {62, 112}, {63, 113}, {64, 114}, {65, 115}, {66,
116}, {67, 116}, {68, 117}, {69, 118}, {70, 118}, {71, 119}}, {{59,
109}, {60, 110}, {61, 110}, {62, 111}, {63, 111}, {64, 112}, {65,
113}, {66, 114}, {67, 115}, {68, 115}, {69, 116}, {70, 117}, {71,
117}, {72, 118}}, {{59, 108}, {60, 109}, {61, 109}, {62, 110}, {63,
110}, {64, 111}, {65, 112}, {66, 113}, {67, 114}, {68, 114}, {69,
115}, {70, 116}, {71, 116}, {72, 117}, {72, 118}, {73, 119}, {74,
120}, {75, 120}, {76, 121}, {77, 121}, {78, 122}}, {{60, 107}, {61,
108}, {62, 108}, {63, 109}, {64, 109}, {65, 110}, {66, 111}, {67,
112}, {68, 113}, {69, 113}, {70, 114}, {71, 115}, {72, 115}, {73,
116}, {73, 117}, {74, 118}, {75, 119}, {76, 119}, {77, 120}, {78,
120}, {79, 121}}, {{61, 106}, {62, 107}, {63, 107}, {64, 108}, {65,
108}, {66, 109}, {67, 110}, {67, 111}, {68, 112}, {69, 112}, {70,
113}, {71, 114}, {72, 114}, {73, 115}, {73, 116}, {74, 117}, {75,
118}, {76, 118}, {77, 119}, {78, 119}, {79, 120}, {80, 122}, {81,
122}, {82, 123}, {83, 123}, {84, 124}, {85, 125}, {86, 125}}, {{61,
105}, {62, 106}, {63, 106}, {64, 107}, {65, 107}, {66, 108}, {67,
109}, {68, 110}, {69, 111}, {70, 111}, {71, 112}, {72, 113}, {73,
113}, {74, 114}, {74, 115}, {75, 116}, {76, 117}, {77, 117}, {78,
118}, {79, 118}, {80, 119}, {81, 121}, {82, 121}, {83, 122}, {84,
122}, {85, 123}, {86, 124}, {87, 124}}, {{62, 104}, {63, 105}, {64,
105}, {65, 106}, {66, 106}, {67, 107}, {68, 108}, {68, 109}, {69,
110}, {70, 110}, {71, 111}, {72, 112}, {73, 112}, {74, 113}, {74,
114}, {75, 115}, {76, 116}, {77, 116}, {78, 117}, {79, 117}, {80,
118}, {81, 120}, {82, 120}, {83, 121}, {84, 121}, {85, 122}, {86,
123}, {87, 123}, {87, 125}, {88, 125}, {89, 126}, {90, 127}, {91,
127}, {92, 128}, {93, 128}}, {{62, 103}, {63, 104}, {64, 104}, {65,
105}, {66, 105}, {67, 106}, {68, 107}, {69, 108}, {70, 109}, {71,
109}, {72, 110}, {73, 111}, {74, 111}, {75, 112}, {75, 113}, {76,
114}, {77, 115}, {78, 115}, {79, 116}, {80, 116}, {81, 117}, {82,
119}, {83, 119}, {84, 120}, {85, 120}, {86, 121}, {87, 122}, {88,
122}, {88, 124}, {89, 124}, {90, 125}, {91, 126}, {92, 126}, {93,
127}, {94, 127}}, {{63, 102}, {64, 103}, {65, 103}, {66, 104}, {67,
104}, {68, 105}, {69, 106}, {70, 107}, {71, 108}, {72, 108}, {73,
109}, {74, 110}, {75, 110}, {76, 111}, {76, 112}, {77, 113}, {78,
114}, {79, 114}, {80, 115}, {81, 115}, {82, 116}, {82, 118}, {83,
118}, {84, 119}, {85, 119}, {86, 120}, {87, 121}, {88, 121}, {88,
123}, {89, 123}, {90, 124}, {91, 125}, {92, 125}, {93, 126}, {94,
126}, {95, 128}, {96, 129}, {97, 129}, {98, 130}, {99, 131}, {100,
131}}, {{64, 101}, {65, 102}, {66, 102}, {67, 103}, {68, 103}, {69,
104}, {70, 105}, {70, 106}, {71, 107}, {72, 107}, {73, 108}, {74,
109}, {75, 109}, {76, 110}, {76, 111}, {77, 112}, {78, 113}, {79,
113}, {80, 114}, {81, 114}, {82, 115}, {83, 117}, {84, 117}, {85,
118}, {86, 118}, {87, 119}, {88, 120}, {89, 120}, {89, 122}, {90,
122}, {91, 123}, {92, 124}, {93, 124}, {94, 125}, {95, 125}, {96,
127}, {97, 128}, {98, 128}, {99, 129}, {100, 130}, {101,
130}}, {{64, 100}, {65, 101}, {66, 101}, {67, 102}, {68, 102}, {69,
103}, {70, 104}, {71, 105}, {72, 106}, {73, 106}, {74, 107}, {75,
108}, {76, 108}, {77, 109}, {77, 110}, {78, 111}, {79, 112}, {80,
112}, {81, 113}, {82, 113}, {83, 114}, {83, 116}, {84, 116}, {85,
117}, {86, 117}, {87, 118}, {88, 119}, {89, 119}, {90, 121}, {91,
121}, {92, 122}, {93, 123}, {94, 123}, {95, 124}, {96, 124}, {96,
126}, {97, 127}, {98, 127}, {99, 128}, {100, 129}, {101,
129}, {101, 131}, {102, 131}, {103, 132}, {104, 133}, {105,
133}, {106, 134}, {107, 134}, {109, 135}, {110, 136}, {111,
136}, {112, 137}, {113, 137}, {114, 138}, {115, 139}}, {{65,
99}, {66, 100}, {67, 100}, {68, 101}, {69, 101}, {70, 102}, {71,
103}, {71, 104}, {72, 105}, {73, 105}, {74, 106}, {75, 107}, {76,
107}, {77, 108}, {78, 109}, {79, 110}, {80, 111}, {81, 111}, {82,
112}, {83, 112}, {84, 113}, {84, 115}, {85, 115}, {86, 116}, {87,
116}, {88, 117}, {89, 118}, {90, 118}, {90, 120}, {91, 120}, {92,
121}, {93, 122}, {94, 122}, {95, 123}, {96, 123}, {97, 125}, {98,
126}, {99, 126}, {100, 127}, {101, 128}, {102, 128}, {102,
130}, {103, 130}, {104, 131}, {105, 132}, {106, 132}, {107,
133}, {108, 133}, {110, 134}, {111, 135}, {112, 135}, {113,
136}, {114, 136}, {115, 137}, {116, 138}}, {{65, 98}, {66,
99}, {67, 99}, {68, 100}, {69, 100}, {70, 101}, {71, 102}, {72,
103}, {73, 104}, {74, 104}, {75, 105}, {76, 106}, {77, 106}, {78,
107}, {78, 108}, {79, 109}, {80, 110}, {81, 110}, {82, 111}, {83,
111}, {84, 112}, {85, 114}, {86, 114}, {87, 115}, {88, 115}, {89,
116}, {90, 117}, {91, 117}, {91, 119}, {92, 119}, {93, 120}, {94,
121}, {95, 121}, {96, 122}, {97, 122}, {97, 124}, {98, 125}, {99,
125}, {100, 126}, {101, 127}, {102, 127}, {102, 129}, {103,
129}, {104, 130}, {105, 131}, {106, 131}, {107, 132}, {108,
132}, {110, 133}, {111, 134}, {112, 134}, {113, 135}, {114,
135}, {115, 136}, {116, 137}, {116, 138}, {117, 139}, {118,
139}, {119, 140}, {120, 141}, {121, 141}, {122, 142}}, {{66,
97}, {67, 98}, {68, 98}, {69, 99}, {70, 99}, {71, 100}, {72,
101}, {72, 102}, {73, 103}, {74, 103}, {75, 104}, {76, 105}, {77,
105}, {78, 106}, {79, 107}, {80, 108}, {81, 109}, {82, 109}, {83,
110}, {84, 110}, {85, 111}, {85, 113}, {86, 113}, {87, 114}, {88,
114}, {89, 115}, {90, 116}, {91, 116}, {92, 118}, {93, 118}, {94,
119}, {95, 120}, {96, 120}, {97, 121}, {98, 121}, {98, 123}, {99,
124}, {100, 124}, {101, 125}, {102, 126}, {103, 126}, {103,
128}, {104, 128}, {105, 129}, {106, 130}, {107, 130}, {108,
131}, {109, 131}, {111, 132}, {112, 133}, {113, 133}, {114,
134}, {115, 134}, {116, 135}, {117, 136}, {117, 137}, {118,
138}, {119, 138}, {120, 139}, {121, 140}, {122, 140}, {123,
141}}, {{67, 96}, {68, 97}, {69, 97}, {70, 98}, {71, 98}, {72,
99}, {73, 100}, {73, 101}, {74, 102}, {75, 102}, {76, 103}, {77,
104}, {78, 104}, {79, 105}, {79, 106}, {80, 107}, {81, 108}, {82,
108}, {83, 109}, {84, 109}, {85, 110}, {86, 112}, {87, 112}, {88,
113}, {89, 113}, {90, 114}, {91, 115}, {92, 115}, {92, 117}, {93,
117}, {94, 118}, {95, 119}, {96, 119}, {97, 120}, {98, 120}, {99,
122}, {100, 123}, {101, 123}, {102, 124}, {103, 125}, {104,
125}, {104, 127}, {105, 127}, {106, 128}, {107, 129}, {108,
129}, {109, 130}, {110, 130}, {111, 131}, {112, 132}, {113,
132}, {114, 133}, {115, 133}, {116, 134}, {117, 135}, {117,
136}, {118, 137}, {119, 137}, {120, 138}, {121, 139}, {122,
139}, {123, 140}, {124, 141}, {125, 142}, {126, 143}, {127,
143}, {128, 144}, {129, 144}, {130, 145}}, {{67, 95}, {68,
96}, {69, 96}, {70, 97}, {71, 97}, {72, 98}, {73, 99}, {73,
100}, {74, 101}, {75, 101}, {76, 102}, {77, 103}, {78, 103}, {79,
104}, {80, 105}, {81, 106}, {82, 107}, {83, 107}, {84, 108}, {85,
108}, {86, 109}, {86, 111}, {87, 111}, {88, 112}, {89, 112}, {90,
113}, {91, 114}, {92, 114}, {93, 116}, {94, 116}, {95, 117}, {96,
118}, {97, 118}, {98, 119}, {99, 119}, {99, 121}, {100, 122}, {101,
122}, {102, 123}, {103, 124}, {104, 124}, {104, 126}, {105,
126}, {106, 127}, {107, 128}, {108, 128}, {109, 129}, {110,
129}, {112, 130}, {113, 131}, {114, 131}, {115, 132}, {116,
132}, {117, 133}, {118, 134}, {118, 135}, {119, 136}, {120,
136}, {121, 137}, {122, 138}, {123, 138}, {124, 139}, {125,
140}, {126, 141}, {127, 142}, {128, 142}, {129, 143}, {130,
143}, {131, 144}}, {{68, 94}, {69, 95}, {70, 95}, {71, 96}, {72,
96}, {73, 97}, {74, 98}, {74, 99}, {75, 100}, {76, 100}, {77,
101}, {78, 102}, {79, 102}, {80, 103}, {81, 104}, {82, 105}, {83,
106}, {84, 106}, {85, 107}, {86, 107}, {87, 108}, {87, 110}, {88,
110}, {89, 111}, {90, 111}, {91, 112}, {92, 113}, {93, 113}, {93,
115}, {94, 115}, {95, 116}, {96, 117}, {97, 117}, {98, 118}, {99,
118}, {100, 120}, {101, 121}, {102, 121}, {103, 122}, {104,
123}, {105, 123}, {105, 125}, {106, 125}, {107, 126}, {108,
127}, {109, 127}, {110, 128}, {111, 128}, {112, 129}, {113,
130}, {114, 130}, {115, 131}, {116, 131}, {117, 132}, {118,
133}, {119, 134}, {120, 135}, {121, 135}, {122, 136}, {123,
137}, {124, 137}, {125, 138}, {125, 139}, {126, 140}, {127,
141}, {128, 141}, {129, 142}, {130, 142}, {131, 143}}, {{68,
93}, {69, 94}, {70, 94}, {71, 95}, {72, 95}, {73, 96}, {74,
97}, {75, 98}, {76, 99}, {77, 99}, {78, 100}, {79, 101}, {80,
101}, {81, 102}, {81, 103}, {82, 104}, {83, 105}, {84, 105}, {85,
106}, {86, 106}, {87, 107}, {88, 109}, {89, 109}, {90, 110}, {91,
110}, {92, 111}, {93, 112}, {94, 112}, {94, 114}, {95, 114}, {96,
115}, {97, 116}, {98, 116}, {99, 117}, {100, 117}, {100,
119}, {101, 120}, {102, 120}, {103, 121}, {104, 122}, {105,
122}, {106, 124}, {107, 124}, {108, 125}, {109, 126}, {110,
126}, {111, 127}, {112, 127}, {113, 128}, {114, 129}, {115,
129}, {116, 130}, {117, 130}, {118, 131}, {119, 132}, {119,
133}, {120, 134}, {121, 134}, {122, 135}, {123, 136}, {124,
136}, {125, 137}, {126, 138}, {127, 139}, {128, 140}, {129,
140}, {130, 141}, {131, 141}, {132, 142}}, {{69, 92}, {70,
93}, {71, 93}, {72, 94}, {73, 94}, {74, 95}, {75, 96}, {75,
97}, {76, 98}, {77, 98}, {78, 99}, {79, 100}, {80, 100}, {81,
101}, {82, 102}, {83, 103}, {84, 104}, {85, 104}, {86, 105}, {87,
105}, {88, 106}, {88, 108}, {89, 108}, {90, 109}, {91, 109}, {92,
110}, {93, 111}, {94, 111}, {95, 113}, {96, 113}, {97, 114}, {98,
115}, {99, 115}, {100, 116}, {101, 116}, {101, 118}, {102,
119}, {103, 119}, {104, 120}, {105, 121}, {106, 121}, {107,
123}, {108, 123}, {109, 124}, {110, 125}, {111, 125}, {112,
126}, {113, 126}, {113, 127}, {114, 128}, {115, 128}, {116,
129}, {117, 129}, {118, 130}, {119, 131}, {120, 132}, {121,
133}, {122, 133}, {123, 134}, {124, 135}, {125, 135}, {126,
136}, {126, 137}, {127, 138}, {128, 139}, {129, 139}, {130,
140}, {131, 140}, {132, 141}}, {{70, 91}, {71, 92}, {72, 92}, {73,
93}, {74, 93}, {75, 94}, {76, 95}, {76, 96}, {77, 97}, {78,
97}, {79, 98}, {80, 99}, {81, 99}, {82, 100}, {83, 101}, {84,
102}, {85, 103}, {86, 103}, {87, 104}, {88, 104}, {89, 105}, {89,
107}, {90, 107}, {91, 108}, {92, 108}, {93, 109}, {94, 110}, {95,
110}, {95, 112}, {96, 112}, {97, 113}, {98, 114}, {99, 114}, {100,
115}, {101, 115}, {102, 117}, {103, 118}, {104, 118}, {105,
119}, {106, 120}, {107, 120}, {107, 122}, {108, 122}, {109,
123}, {110, 124}, {111, 124}, {112, 125}, {113, 125}, {114,
126}, {115, 127}, {116, 127}, {117, 128}, {118, 128}, {119,
129}, {120, 130}, {120, 131}, {121, 132}, {122, 132}, {123,
133}, {124, 134}, {125, 134}, {126, 135}, {127, 136}, {128,
137}, {129, 138}, {130, 138}, {131, 139}, {132, 139}, {133,
140}}, {{70, 90}, {71, 91}, {72, 91}, {73, 92}, {74, 92}, {75,
93}, {76, 94}, {76, 95}, {77, 96}, {78, 96}, {79, 97}, {80,
98}, {81, 98}, {82, 99}, {83, 100}, {84, 101}, {85, 102}, {86,
102}, {87, 103}, {88, 103}, {89, 104}, {89, 106}, {90, 106}, {91,
107}, {92, 107}, {93, 108}, {94, 109}, {95, 109}, {96, 111}, {97,
111}, {98, 112}, {99, 113}, {100, 113}, {101, 114}, {102,
114}, {102, 116}, {103, 117}, {104, 117}, {105, 118}, {106,
119}, {107, 119}, {108, 121}, {109, 121}, {110, 122}, {111,
123}, {112, 123}, {113, 124}, {114, 124}, {114, 125}, {115,
126}, {116, 126}, {117, 127}, {118, 127}, {119, 128}, {120,
129}, {121, 130}, {122, 131}, {123, 131}, {124, 132}, {125,
133}, {126, 133}, {127, 134}, {127, 135}, {128, 136}, {129,
137}, {130, 137}, {131, 138}, {132, 138}, {133, 139}}, {{71,
89}, {72, 90}, {73, 90}, {74, 91}, {75, 91}, {76, 92}, {77,
93}, {77, 94}, {78, 95}, {79, 95}, {80, 96}, {81, 97}, {82,
97}, {83, 98}, {84, 99}, {85, 100}, {86, 101}, {87, 101}, {88,
102}, {89, 102}, {90, 103}, {90, 105}, {91, 105}, {92, 106}, {93,
106}, {94, 107}, {95, 108}, {96, 108}, {97, 110}, {98, 110}, {99,
111}, {100, 112}, {101, 112}, {102, 113}, {103, 113}, {103,
115}, {104, 116}, {105, 116}, {106, 117}, {107, 118}, {108,
118}, {109, 120}, {110, 120}, {111, 121}, {112, 122}, {113,
122}, {114, 123}, {115, 123}, {115, 124}, {116, 125}, {117,
125}, {118, 126}, {119, 126}, {120, 127}, {121, 128}, {122,
129}, {123, 130}, {124, 130}, {125, 131}, {126, 132}, {127,
132}, {128, 133}, {128, 134}, {129, 135}, {130, 136}, {131,
136}, {132, 137}, {133, 137}, {134, 138}}, {{71, 88}, {72,
89}, {73, 89}, {74, 90}, {75, 90}, {76, 91}, {77, 92}, {77,
93}, {78, 94}, {79, 94}, {80, 95}, {81, 96}, {82, 96}, {83,
97}, {84, 98}, {85, 99}, {86, 100}, {87, 100}, {88, 101}, {89,
101}, {90, 102}, {90, 104}, {91, 104}, {92, 105}, {93, 105}, {94,
106}, {95, 107}, {96, 107}, {97, 109}, {98, 109}, {99, 110}, {100,
111}, {101, 111}, {102, 112}, {103, 112}, {103, 114}, {104,
115}, {105, 115}, {106, 116}, {107, 117}, {108, 117}, {109,
119}, {110, 119}, {111, 120}, {112, 121}, {113, 121}, {114,
122}, {115, 122}, {115, 123}, {116, 124}, {117, 124}, {118,
125}, {119, 125}, {120, 126}, {121, 127}, {122, 128}, {123,
129}, {124, 129}, {125, 130}, {126, 131}, {127, 131}, {128,
132}, {128, 133}, {129, 134}, {130, 135}, {131, 135}, {132,
136}, {133, 136}, {134, 137}}, {{72, 87}, {73, 88}, {74, 88}, {75,
89}, {76, 89}, {77, 90}, {78, 91}, {78, 92}, {79, 93}, {80,
93}, {81, 94}, {82, 95}, {83, 95}, {84, 96}, {85, 97}, {86,
98}, {87, 99}, {88, 99}, {89, 100}, {90, 100}, {91, 101}, {91,
103}, {92, 103}, {93, 104}, {94, 104}, {95, 105}, {96, 106}, {97,
106}, {98, 108}, {99, 108}, {100, 109}, {101, 110}, {102,
110}, {103, 111}, {104, 111}, {104, 113}, {105, 114}, {106,
114}, {107, 115}, {108, 116}, {109, 116}, {110, 118}, {111,
118}, {112, 119}, {113, 120}, {114, 120}, {115, 121}, {116,
121}, {116, 122}, {117, 123}, {118, 123}, {119, 124}, {120,
124}, {121, 125}, {122, 126}, {123, 127}, {124, 128}, {125,
128}, {126, 129}, {127, 130}, {128, 130}, {129, 131}, {129,
132}, {130, 133}, {131, 134}, {132, 134}, {133, 135}, {134,
135}, {135, 136}}}

• why not delete {1,2} also from first row since that show in third row? Let me make sure I understand. You want at end each row to have unique elements that do not show in any other row? Nov 23, 2021 at 0:05

ClearAll[deleteDupes1]
deleteDupes1 = Module[{f}, f[x_] := (f[x] = Sequence[]; x); Map[f, #, {2}]] &;


Examples:

lists = {{{1, 2}, {3, 4}},
{{5, 6}, {7, 8}, {25, -1}, {-14, -3}},
{{1, 2}},
{{5, 6}, {7, 8}, {22, -1}, {-11, -3}}};

deleteDupes1 @ lists // Column


To avoid (to the extent possible) empty sets in the output we can use:

ClearAll[deleteDupes2]
deleteDupes2 = deleteDupes1[Sort @ #][[Ordering @ Ordering @ #]] &;

deleteDupes2 @ lists // Column


We get the same result using deleteDupes3 instead of deleteDupes2:

ClearAll[deleteDupes3]
deleteDupes3 = Permute[deleteDupes1[Sort @ #], Ordering @ #] &;

deleteDupes3[lists] // Column


I do not quite understand why you did not delete {1,2} also, since you said you want to delete duplicates.

If this is really what you want, here is one possible way.

(mat = {{{1, 2}, {3, 4}}, {{5, 6}, {7, 8}, {25, -1}, {-14, -3}}, {{1,
2}}, {{5, 6}, {7, 8}, {22, -1}, {-11, -3}}}) // MatrixForm


Now

Do[Do[
currentRow = mat[[n]];
mat[[n]] = DeleteCases[currentRow, Alternatives @@ mat[[m]]]
,{m, n + 1, Length[mat]}],{n, 1, Length[mat] - 1}
];


And now

MatrixForm[mat]


This goes over reach row at a time, and scan all rows below it, deletes duplicates in current row as it finds them.

This is k*O(n^2) where n is number of row in the matrix where k is some large number. May be there is faster way. Running this on your test data took only 0.00462061 seconds, using RepeatedTiming.

• I am so so sorry @Nasser, I wanted to delete {1,2}, I did not see it was a duplicate, Thank you for your answer and vigilance. I edited my question Nov 23, 2021 at 10:26