# DeleteDuplicates without destroying SparseArray

When I use DeleteDuplicates to delete equal rows of a sparse array, the output is not a sparse array anymore. In my application, the sparse arrays are huge but very sparse, so converting back and forth is quite annoying.

Is there a version of, or alternative to, DeleteDuplicates that always preserves sparse arrays? (i.e. without converting back and forth between dense and sparse representations)

• You mean, you want to delete duplicate rows? Commented Jun 16, 2023 at 12:00
• Yes. But preferably still with the possibility to use a test function of choice (e.g. when two rows are equal up to sign)
– Gert
Commented Jun 16, 2023 at 13:43
• You can avoid unsparsifying rows by doing SparseArray @ DeleteDuplicates[List @@ sparse] Commented Jun 16, 2023 at 14:23
• Deleting duplicate rows would change the dimensions of the array. That in turn would change the coordinate->value rules. You won't be preserving much of anything. So, if DeleteDuplicates did return a sparse array, you'd just have a whole other set of annoyances to deal with. Are you sure DeleteDuplicates is the operation you want? Maybe there is a more algebraic operation that you're looking for. Commented Jun 16, 2023 at 15:44

## 1 Answer

It often helps when the OP provides their own test case. But here goes, assuming any test case sufficiently illustrates the problem as we are implicitly invited to do:

Quit[]

SeedRandom[0];
sparse = SparseArray[
Thread[RandomInteger[{1, 1000}, {100, 2}] -> 1], {1000, 1000}]

MaxMemoryUsed[]

(*  150671984  *)

sparse[[DeleteDuplicatesBy[Range@Length@sparse, sparse[[#]] &]]]

MaxMemoryUsed[]

(*  150671984  *)


Or per the OP's comment under the OP:

SeedRandom[0];
sparse = SparseArray[
Thread[RandomInteger[{1, 1000}, {100, 2}] ->
RandomChoice[{-1, 1}, 100]], {1000, 1000}]

sparse[[
DeleteDuplicatesBy[Range@Length@sparse, Abs[sparse[[#]]] &]]]


(Same dimensions since Abs[-1] == 1.

• Somewhat inspired by this answer to a question of mine. Which I guess I still remember. Commented Jun 17, 2023 at 23:15