# How to get list of duplicates when using DeleteDuplicates?

This might be easy, but can't find a way to use DeleteDuplicates to get also list of the actual duplicates.

Example:

lstA = {1, 2, 4, 4, 6, 7, 8, 8};
r = DeleteDuplicates[lstA]
(* {1, 2, 4, 6, 7, 8} *)


I also wanted to get list of the actual duplicates, which are {4,8} in this example. It would have been nice if DeleteDuplicates would also return those, but there is no option there for that. I do not know what many Mathematica functions do not return back more useful information when called. Many seem to return one piece of information only, and one has to call another function to get another piece of information.

For example, here DeleteDuplicates could had an API like this

{r,d}=DeleteDuplicates[lst]


and r will contain the list after duplicates are moved, but d would contain the actual list of duplicates. To make it even more useful, it can be

{r,d,p}=DeleteDuplicates[lst]


Where p will be the positions of the duplicates in the original list. Matlab seems to do it this way. Many functions there can return more than one piece of information at a time. This might be due to WL being functional programming language, and designed for cascading function calls, where each function only does one thing at a time. I am not sure now.

DeleteDuplicates.html

• You want to get list of the duplicates in general, or such list with DeleteDuplicates usage?
– Kuba
Commented Nov 28, 2013 at 8:34
• @Kuba actually I found couple of places where they talk about finding positions of duplicates in lists but did not yet understand the code. I was hoping that DeleteDuplicate can do it. But if you know of an easy way to get list of duplicates (without using DeleteDuplicates) that will work also. Thanks. Commented Nov 28, 2013 at 8:51
• What's with the wolfram-language tag? Is it necessary? In principle, this tag could be applied to every question... Commented Nov 28, 2013 at 12:03
• Sorry if I sounded harsh, but I really don't share SW's zeal about all the "new" products. And as I gathered it, Wolfram Language is really not much more than (the core of) Mathematica, so I guess we don't want to go down the slope where we end up retagging all questions here and/or on SO, or even renaming Mathematica.SE to Wolfram.SE. Personally I find it bad style to name a product and a language after yourself. Commented Nov 29, 2013 at 8:00
• @IstvánZachar I agree. This tag is now banned at the system level and can no longer be used. Please also be on the lookout for variations of "wolfram-language".
– rm -rf
Commented Dec 1, 2013 at 16:28

Perhaps one of the simplest ways is to use Tally:

p = {1, 2, 4, 4, 6, 7, 8, 8};

Cases[Tally @ p, {x_, n_ /; n > 1} :> x]

{4, 8}


A somewhat faster formulation:

Pick[#, Unitize[#2 - 1], 1] & @@ Transpose[Tally @ p]


Taking the optimization to a rather excessive degree:

#[[SparseArray[#2, Automatic, 1]["AdjacencyLists"]]] & @@ Transpose[Tally @ p]


Though not as fast as the SparseArray optimized form of Tally, an alternative is to use Split after sorting. This is reasonably clean and fast:

Flatten[Split @ Sort @ p, {2}][[2]]

{4, 8}


For Integer data this method is twice as fast as any other listed here:

With[{s = Sort @ p},
DeleteDuplicates @
s[[ SparseArray[Unitize @ Differences @ s, Automatic, 1]["AdjacencyLists"] ]]
]


## Timings

p = RandomInteger[1*^8, 1*^6];

Cases[Tally @ p, {x_, n_ /; n > 1} :> x] // Timing // First

Pick[#, Unitize[#2 - 1], 1] & @@ Transpose[Tally @ p] // Timing // First

#[[SparseArray[#2, Automatic, 1]["AdjacencyLists"]]] & @@ Transpose[Tally @ p] //
Timing // First

Flatten[Split @ Sort @ p, {2}][[2]] // Timing // First

With[{s = Sort @ p},
DeleteDuplicates @
s[[ SparseArray[Unitize @ Differences @ s, Automatic, 1]["AdjacencyLists"] ]]
] // Timing // First

0.827

0.343

0.265

0.343

0.11

• @Nasser & Mr.Wizard I like this answer but it is not the final answer to the question. You havn't asked about list of duplicates but about that and standard result from DD at once. I don't mean that something terrible is going on here but it is strange that the accepted answer do not fit the question, isn't it? :)
– Kuba
Commented Nov 29, 2013 at 7:44
• I agree with @Kuba, as if one omits the request for the advanced API the question becomes quite trivial, perhaps even would have been closed down due to the fact that the answer is easy to find in the documentation. Nevertheless, I (and others no doubt) find the Q inspiring, as it forced me to explore Reap in more depths. Commented Nov 29, 2013 at 7:51
• Kuba and @István I didn't see the point in packaging this into a function that returns "{r, d, p}". Methods for the positions of duplicates can be found here. If you are focusing on the "API" I believe that has been asked before, perhaps by rm -rf. I'll try to find it. Commented Nov 29, 2013 at 12:26
• Commented Nov 29, 2013 at 12:41
• The difference is that the result in form {r, d, p} is more challenging in terms of efficiency. For example in my myClone positions are gathered with additional scanning, where one could get them earlier if knows how.
– Kuba
Commented Nov 29, 2013 at 12:58

If you have to use DeleteDuplicates you can use Sow/Reap:

{#, Pick @@ Transpose[GatherBy[#2[[1]]][[;; , 1]]]} & @@ Reap[
DeleteDuplicates[lstA, (Sow[{#1, SameQ[##]}]; SameQ[##]) &]]

{{1, 2, 4, 6, 7, 8}, {4, 8}}


Here's more general and faster approach:

myClone[list_, test_: Identity] := Composition[
{#[[1]], #[[2]], Position[list, #] & /@ #[[2]]} &,
{#[[;; , 1]], DeleteCases[#, {_}][[;; , 1]]} &,
GatherBy[#, test] &
][list]

lstA = RandomInteger[10, 10]
myClone[lstA]

{5, 6, 3, 5, 10, 10, 8, 7, 0, 10}

{{5, 6, 3, 10, 8, 7, 0}, {5, 10}, {{{1}, {4}}, {{5}, {6}, {10}}}}

• Nice. Now would it not be good if Mathematica DeleteDuplicate returned both of these lists as well? (if user wanted) Commented Nov 28, 2013 at 8:53
• @Nasser I don't know, I think it might be the case where there is no "most useful output" so they left it in quite raw form :)
– Kuba
Commented Nov 28, 2013 at 9:00

Similar to Kuba's solution, but more compact, using Reap's third argument to remove duplicates from the duplicates:

x = {1, 2, 3, 4, 5, 4, 4, 4, 5, 1};

Reap[DeleteDuplicates[x, If[SameQ[##], Sow@#1; True, False] &], _, Union@#2 &]

 {{1, 2, 3, 4, 5}, {{1, 4, 5}}}


I am posting a second answer because this is a different method unrelated to the first.
I wondered how I might approach this if Tally did not exist. I came up with using Ordering on a reverse-sorted list as a way to look for duplicates. It seems to work, and I think it's fairly interesting. By nature it sorts the list of duplicates rather than giving them in order of appearance as Tally does.

duplicates[p_] :=
With[{sp = Sort @ p}, sp[[
SparseArray[Unitize[1 - Differences @ Ordering @ Reverse @ sp], Automatic, 1]
]]
] // DeleteDuplicates

duplicates[{1, 2, 4, 4, 6, 7, 8, 8}]

{4, 8}


It is competitively fast compared to my first answer:

p = RandomInteger[8*^6, 2*^6];

Cases[Tally@p, {x_, n_ /; n > 1} :> x] // Length // Timing

Pick[#, Unitize[#2 - 1], 1] & @@ Transpose[Tally@p] // Length // Timing

#[[SparseArray[#2, Automatic, 1]["AdjacencyLists"]]] & @@ Transpose[Tally@p] //
Length // Timing

duplicates[p] // Length // Timing

{1.70041, 212277}

{0.592804, 212277}

{0.608404, 212277}

{0.358802, 212277}


On lists with extreme duplication it is a bit slower:

p = RandomInteger[1*^6, 5*^6];

(* same timing code as before *)

{1.57561, 959792}

{0.904806, 959792}

{0.904806, 959792}

{0.982806, 959792}


Important: Union with one argument does exactly what I describe below. I don't know how I missed that for so long. Anyway I suppose it is still nice to be able to return a list of the positions of the duplicates. And there may be other cases where you know a result is sorted I guess.

I have long been a bit frustrated about the fact that there is neither an option for Sort that deletes duplicates, nor an option for DeleteDuplicates that makes it assume its input is sorted. DeleteDuplicates probably first sorts its argument, but still even determining that a list is sorted is unnecessary overhead. I saw that in your question the numbers were sorted, so I have focussed on this case.

In my opinion, your questions and my own made for a good case to make a library with LibraryLink. Here is the code. It is a bit long, especially because similar code is used over and over.

#include "WolframLibrary.h"

static MTensor positions;
// static MTensor frequencies;

/* Return the version of Library Link */
DLLEXPORT mint WolframLibrary_getVersion( ) {
return WolframLibraryVersion;
}

/* Initialize Library */
DLLEXPORT int WolframLibrary_initialize( WolframLibraryData libData) {
return LIBRARY_NO_ERROR;
}

/* Uninitialize Library */
DLLEXPORT void WolframLibrary_uninitialize( WolframLibraryData libData) {
return;
}

//DD is for delete duplicates
DLLEXPORT int sortedDD_T_T(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
MTensor input; //not redundant as we access this two times, once for data and once for dimension
MTensor result;

int err = LIBRARY_NO_ERROR;

mint *inputDataPtr;
mint *resultDataPtr;

mint inputDataLen;

input = MArgument_getMTensor(Args[0]);

inputDataPtr = libData->MTensor_getIntegerData(input);

inputDataLen = * libData->MTensor_getDimensions(input);

mint array[inputDataLen];
mint *arrayPtr;
arrayPtr = array;

//below is basically the first (0th) iteration
mint max = *inputDataPtr;
inputDataPtr++;
*arrayPtr = max;
arrayPtr++;

mint val;
mint resultDataLenCounter = 1;
mint iiii;

for(iiii = 1; iiii < inputDataLen; iiii++)
{
val = *inputDataPtr;
inputDataPtr++;
if( val != max)
{
*arrayPtr = val;
arrayPtr++;
resultDataLenCounter++;
max = val;
}
}

err = libData->MTensor_new(MType_Integer, 1, &resultDataLenCounter, &result);
if(err) goto error_label;

resultDataPtr = libData->MTensor_getIntegerData(result);

arrayPtr = array;

for(iiii = 0; iiii < resultDataLenCounter; iiii++){
*(resultDataPtr++) = * (arrayPtr++);
}

libData->MTensor_disown(input);

MArgument_setMTensor(Res, result);
return err;

error_label:
libData->MTensor_disown(input);

if(!err) err = 1;
return err;

}

//WP is for with positions

DLLEXPORT int sortedDDWP_T_T(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
MTensor input; //not redundant as we access this two times, once for data and once for dimension
MTensor result;

int err = LIBRARY_NO_ERROR;

mint *inputDataPtr;
mint *resultDataPtr;
mint *posDataPtr;

mint inputDataLen;

input = MArgument_getMTensor(Args[0]);

inputDataPtr = libData->MTensor_getIntegerData(input);

inputDataLen = * libData->MTensor_getDimensions(input);

mint uqArray[inputDataLen];
mint *uqArrayPtr;
uqArrayPtr = uqArray;

mint posArray[inputDataLen];
mint *posArrayPtr;
posArrayPtr = posArray;

mint posLen;

//below is basically the first (0th) iteration
mint max = *inputDataPtr;
inputDataPtr++;
*uqArrayPtr = max;
uqArrayPtr++;

mint val;
mint resultDataLenCounter = 1;
mint iiii;

for(iiii = 1; iiii < inputDataLen; iiii++)
{
val = *inputDataPtr;
inputDataPtr++;
if( val != max)
{
*uqArrayPtr = val;
uqArrayPtr++;
resultDataLenCounter++;
max = val;
}
else
{
*posArrayPtr = iiii;
posArrayPtr++;
}
}

posLen = inputDataLen - resultDataLenCounter;

err = libData->MTensor_new(MType_Integer, 1, &resultDataLenCounter, &result);
if(err) goto error_label;

err = libData->MTensor_new(MType_Integer, 1, &posLen, &positions);
if(err) goto error_label;

resultDataPtr = libData->MTensor_getIntegerData(result);
posDataPtr = libData->MTensor_getIntegerData(positions);

uqArrayPtr = uqArray;
posArrayPtr = posArray;

for(iiii = 0; iiii < resultDataLenCounter; iiii++){
*(resultDataPtr++) = * (uqArrayPtr++);
}

for(iiii = 0; iiii < posLen; iiii++){
*(posDataPtr++) = *(posArrayPtr++);
}

libData->MTensor_disown(input);

MArgument_setMTensor(Res, result);
return err;

error_label:
libData->MTensor_disown(input);

if(!err) err = 1;
return err;

}

DLLEXPORT int getPositions(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {

MArgument_setMTensor(Res, positions);
// libData->MTensor_disown(positions);
return 0;
}


(* libraryName = "libraryName"; *) (*this is the last line from the "instructions"*)


The following code loads the functions.

SortedDD is the function that simply deletes duplicates from a sorted list.

sortedDD =
"sortedDD_T_T", {{Integer, 1, "Shared"}}, {Integer, 1}]


sortedDDWP is the function that makes a list with duplicates deleted, as well of a list of positions of the duplicates.

sortedDDWP =
"sortedDDWP_T_T", {{Integer, 1, "Shared"}}, {Integer, 1}]


As we can only return one argument in LibraryLink, we have to work around this and get the positions with another function call. This is what getPos does.

getPos = LibraryFunctionLoad[libraryName,
"getPositions", {}, {Integer, 1}]


Performance

WARNING: One big problem with my code is that it crashes the kernel for large input. This is because of a stackoverflow that I intend to fix.

Now let's create some random numbers. I chose to generate a BinomialProcess, i.e., a process that increases by with probably 1/2 and otherwise stays the same. So a sequence could be normal = {0,1,1,1,2,2} and we expect about 50% duplicates.

nnnn = 1002000;
randomize :=
Block[{rands},
rands = RandomInteger[1, nnnn];
normal = Accumulate[rands]
];


now let's compare with DeleteDuplicates

sortedDD@normal // Timing // First
DeleteDuplicates@normal // Timing // First


0.008401
0.018445

That is quite nice :). Now let's see what happens if we also generate the positions list

kkkk = 522000;
(
mOut =
{
sortedDDWP[normal[[;; kkkk]]],
getPos[]
};
) // Timing // First


0.006506

and we have

mOut[[1]] == Delete[normal[[;; kkkk]], List /@ mOut[[2]]]


True

If you read the C Code, you will that I make an array, only to copy the contents of that array into another array. Unfortunately I think there is no good way around this.

• Oh by the way to get the actual duplicates just do Part[normal[[;;kkkk]], mOut[[2]]] Commented Dec 2, 2013 at 0:25
• I don't think DeleteDuplicates sorts its argument first: DeleteDuplicates[{5, 4, 3, 4, 1}, (Print[{##}]; SameQ[##]) &] clearly reports otherwise. Commented Dec 2, 2013 at 9:49
• @IstvánZachar it's possible that the behavior is different for a user specified test. I could not find a claim of this for DeleteDuplicates, but here Leonid claims the behavior for Union changes. Commented Dec 2, 2013 at 11:55
• You are right Jacob, similar tests for DeleteDuplicates indicate similar increase in time when SameQ[##]& is used instead of SameQ. Thanks for pointing it out! Commented Dec 2, 2013 at 13:47
• @IstvánZachar :) no problem! Good that you tested it out. However do not underestimate the impact on timing of that SameQ[##]& requires the tests to be evaluated by Mathematica, whereas otherwise everything can be done "in C". I wish my code worked on larger data so we could do more comparisons. The most convincing thing for me is that calling DeleteDuplicates on increasingly large lists seems to give a (quasi-)linear increase in timing. Commented Dec 2, 2013 at 14:02

This uses a function which redefines itself, and MapIndexed to get the position data:

deldup[z_] := Module[{f},
f[x_, _] := (f[x, q_] := (Sow[q, x]; Unevaluated@Sequence[]); x);
Reap[MapIndexed[f, z], _, Rule]]

deldup[{1, 2, 3, 4, 2, 3, 2}]
(* {{1, 2, 3, 4}, {2 -> {{5}, {7}}, 3 -> {{6}}}} *)


Some variants with Reap/Sow or Tally:

fun1[u_] := {#[[1]], Pick[#[[1]], (# > 1) & /@ #[[2]]]} &@
Transpose[Reap[Sow[1, #] & /@ u, _, {#1, Total@#2} &][[2]]]


or

fun2[u_] := {#[[1]], Pick[#[[1]], (# > 1) & /@ #[[2]]]} &@
Transpose@Tally[u]

• I am giving a +1 for using Transpose@Tally[u] but see my answer for a faster method to do the second part. Commented Nov 28, 2013 at 11:12
• @Mr.Wizard thank you for instruction...am always glad to improve...Unitize very nice Commented Nov 29, 2013 at 6:30

Another way to get the dupe-free list and the list of duplicates:

With[{grouped = Gather[lstA, SameQ], pattern = {x_, __} :> x},
{Flatten@Replace[grouped, pattern, 1], Cases[grouped, pattern]}
]
(* {{1, 2, 4, 6, 7, 8}, {4, 8}} *)


delinit[v_List] := Module[{f}, f[x_] := (f[x] = x; Unevaluated[]); f/@v] deletes the initial instance of every term in v. All instances after the first are kept, in the order in which they come. This is the logical complement of DeleteDuplicates.

EDIT - deli does the same thing, but faster.

deli[v_List] := Delete[v,Transpose@List@Part[Range[Length@v][[#]],
Most@FoldList[Plus,1,Length/@Split@v[[#]]]]]& @ Ordering@v

• Nice! I do not know how it works, but it does :) btw, the FE on V 9.01 does not like Unevaluated[] it puts little red marker between the [] but it still worked. Commented Nov 29, 2013 at 7:50
• If you know that v contains no sublists then you could change Unevaluated[] to {} and do Flatten[f/@v]. Or change Unevaluated[] to Hold@Sequence[] and do ReleaseHold[f/@v], which should work no matter what is in v. Commented Nov 29, 2013 at 8:01
• @Nasser - Sorry, I forgot about what's probably the easiest fix -- just change Unevaluated[] to Mr.Wizard's "vanishing function", ##&[]. Commented Nov 30, 2013 at 3:00
• @Nasser you can change the SyntaxInformation of Unevaluated if you like to use the zero or multiple argument form; AFAICT it is perfectly usable that way. Ray, thanks for the mention regarding ##&[] -- it is my favorite method. Commented Nov 30, 2013 at 5:48

For fun of it: list = {1, 2, 4, 4, 8, 4, 6, 7, 8, 8};

DeleteDuplicates@Cases[Subsets[list, {2}], {a_, a_}][[;; , 1]]


{4, 8}

• Not efficient, but fun indeed. +1 :-) Nice use of named patterns, by the way. Commented Feb 16, 2016 at 19:16
CustomDeleteDuplicates[lst_List] :=
Module[{lstA = lst}, {lstA //. {s___, a_, a_, g___} -> {s, a, g},
Complement[lstA, lstA //. {s___, a_, a_, g___} -> {s, 0, g}]}]


{{1, 2, 4, 6, 7, 8}, {4, 8}}

EDIT: As pointed out I have updated code that is working with all use cases. I have used patterns to find out duplicates to filter out as many repetitions as can be made. It will filter out all even pairs and last remaining in case of odd count will be dealt with next function acting on it.

CustomDeleteDuplicates[li_List] :=
Block[{l = li}, {Flatten[l] //. {s___, a_, k___, a_, g___} -> {s, a, k, g},
Complement[l,Pick[(l //. {s___, a_, k___, a_, g___} -> {s, k, g}),
Count[l, #] & /@ (l //. {s___, a_, k___, a_, g___} -> {s, k, g}),  1]]}]


{13, 12, 2, 15, 2, 6, 7, 2, 4, 7, 12, 15, 6, 15, 12, 14, 15, 10, 14, 1}

Gather[l]


{{13}, {12, 12, 12}, {2, 2, 2}, {15, 15, 15, 15}, {6, 6}, {7, 7}, {4}, {14, 14}, {10}, {1}}

CustomDeleteCases[l]

{{13, 12, 2, 15, 6, 7, 4, 14, 10, 1}, {2, 6, 7, 12, 14, 15}}

And an even simpler solution with ReplaceRepeated:

list = {13, 12, 2, 15, 2, 6, 7, 2, 4, 7, 12, 15, 6, 15, 12, 14, 15, 10, 14, 1};
{list, {}} //. {{a___, b_, c___, b_, d___}, {du___}} :> {{a, b, c, d}, Union@{du, b}}

 {{13, 12, 2, 15, 6, 7, 4, 14, 10, 1}, {2, 6, 7, 12, 14, 15}}

• This only works if the duplicates are next to each other, and there are no more than 2 of each. You should also use RuleDelayed (:>) instead of Rule (->) if using named patterns to avoid interference with any predifined s, a or g. Commented Nov 29, 2013 at 10:07
• @IstvánZachar : Actually I knew my mistake, I have updated it now, I think its working fine now. I always had some doubt about -> vs ':>, now I think am more clear on reading your comment. Thanks :) Commented Nov 30, 2013 at 16:43
• I still find it pretty convoluted :) You scan through the lists so many times! Also, the Block is unnecessary, you could work on li directly; Complement and Pick can be replaced by Union; you do the //. three times, though one would be enough. I could not resist and edited in a simpler solution, because I find your ReplaceRepeated approach quite interesting! Please feel free to revert my edit if you think so! Commented Nov 30, 2013 at 18:25
• @IstvánZachar I am always open to anyone editing my answer to make it better so, thanks :) . I tend to use 'Block' to prevent any outside variable intervention, nothing else. Commented Nov 30, 2013 at 20:14
• I see your intention, though li in f[li_]:=(...) is as much local and protected from the outside as l in f[li_]:=Block[{l=li}, (...)]. Block however does not make by itself s, k, g locals. Commented Dec 1, 2013 at 10:10

With Sow/Reap and Pick

listp = {1, 2, 4, 4, 6, 7, 8, 8};


(i) List duplicate values

Reap[Sow[1,listp],_,Pick[#1,Unitize[Length@#2-1],1]&][[2]]


{4, 8}

(ii) Delete all duplicates (see this question)

Reap[Sow[1,listp],_,Pick[#1,#2,{1}]&][[2]]


{1, 2, 6, 7}

(iii) Delete duplicates (from Reap Documentation, unsorted Union)

Reap[Sow[1,listp],_,#1&][[2]]


{1, 2, 4, 6, 7, 8}

list = {4, 1, 2, 4, 4, 6, 7, 8, 8};


List duplicate values

dups = SequenceCases[list, {a_, a_} :> a]


{4, 8}

Delete all duplicates

DeleteElements[list, dups]


{1, 2, 6, 7}

WolframLanguageData[#, {"VersionIntroduced", "DateIntroduced"}] & /@ {"SequenceCases", "DeleteElements"}


With E. Chan-López data

list = {4, 1, 2, 4, 4, 6, 7, 8, 8};


Using Counts

Keys @ Select[GreaterThan @ 1] @ Counts[list]


{4, 8}

Using SequenceCases

SequenceCases[list, {OrderlessPatternSequence[_, a_, a_]} :> a]


{4, 8}

Using SubsetCases and Cases:

list = {4, 1, 2, 4, 4, 6, 7, 8, 8};

Cases[SubsetCases[list, {s_ ..}], x_ /; Length[x] > 1 :> First@x]

(*{4, 8}*)


Or using GroupBy and Repeated:

g = GroupBy[list, Repeated];

Keys@Select[KeyMap[Times @@ List @@ # &, g], Length@# > 1 &]

(*{4, 8}*)


Thanks to @DanielLichtblau for the following alternative using Duplitcates (by Sander Huisman):

Union[ResourceFunction["Duplicates"][list]]

(*{4, 8}*)

• Also: In[2215]:= Union[ResourceFunction["Duplicates"][list]] Out[2215]= {4, 8} Commented Dec 20, 2023 at 20:30
• Thanks for mentioning that it exists as a resource function, @DanielLichtblau! Commented Dec 20, 2023 at 20:38

Probably not working in every case:

Select[Partition[Sort[lstA], 2, 1, 1], #[[1]] == #[[2]] &][[All, 1]]

(* {4, 8} *)


Using PositionIndex:

lstA = {1, 2, 4, 4, 6, 7, 8, 8};
pi = PositionIndex[lstA]


<|1 -> {1}, 2 -> {2}, 4 -> {3, 4}, 6 -> {5}, 7 -> {6}, 8 -> {7, 8}|>

The elements at the First position in each of the Values in pi are the non-duplicates; the Rest are positions with duplicates. Therefore,

{lstA[[KeyValueMap[First@*First][pi]]],
lstA[[Sequence @@@ KeyValueMap[Rest@*First][pi] ]]
}


Result:

{{1, 2, 4, 6, 7, 8}, {4, 8}}