# Filtering the close values from the list? [duplicate]

I found some difficulty to flatten the data.

Let's say we have a list such that,

list={-0.0223, -0.0199, -0.0195, -0.0159, -0.0159, -0.0122, -0.0126}


You can see that [-0.0199, -0.0195], [-0.0159, -0.0159], [-0.0122, -0.0126] are very close, actually these are duplicates from the precision error that I do not need, I only want one value. Maybe Mean? or chose one random value among the duplicates.

(*I want to have*)
{
-0.0223,
Mean[{-0.0199, -0.0195}],
Mean[{-0.0159, -0.0159}],
Mean[{-0.0122, -0.0126}]
}


I tried to use Select, but I was not sure how to do it, anyone can help me? Thank you!

## marked as duplicate by Mr.Wizard♦ list-manipulation StackExchange.ready(function() { if (StackExchange.options.isMobile) return; $('.dupe-hammer-message-hover:not(.hover-bound)').each(function() { var$hover = $(this).addClass('hover-bound'),$msg = $hover.siblings('.dupe-hammer-message');$hover.hover( function() { $hover.showInfoMessage('', { messageElement:$msg.clone().show(), transient: false, position: { my: 'bottom left', at: 'top center', offsetTop: -7 }, dismissable: false, relativeToBody: true }); }, function() { StackExchange.helpers.removeMessages(); } ); }); }); Feb 19 '16 at 21:57

You could try DeleteDuplicates with a custom test:

input = {-0.0223, -0.0199, -0.0195, -0.0159, -0.0159, -0.0122, -0.0126};

DeleteDuplicates[input, (Abs[#1 - #2] < 10^-3 &)]

(* ==> {-0.0223, -0.0199, -0.0159, -0.0122} *)


Edit

In the question, it's not completely clear what the nature of the "precision error" is. If it's actually an absolute error that makes each element of the list uncertain, then it may actually be better (and much faster) to use Round:

error = 9. 10^-4;
DeleteDuplicates[Round[input, error]]
(* ==> {-0.0225, -0.0198, -0.0162, -0.0126} *)


So I defined an error and rounded the input to multiples of this error before doing DeleteDuplicates. This of course returns the rounded values, reflecting the fact that each entry shouldn't be specified to higher precision than multiples of the base error in the first place.

You can accomplish the averaging of the nearly equal successive elements (while not averaging those that are far apart) using the MeanShiftFilter function:

DeleteDuplicates@MeanShiftFilter[list, 2, 0.001]
{-0.0223, -0.0197, -0.0159, -0.0124}


I believe this version may be relatively fast:

Mean /@ Split[Sort@list, Abs[#1 - #2] < 10^-3 &]


One problem I see with this method (and perhaps the Gather method too?) is that this accepts runs of consecutive elements where the adjacent elements are indistinguishable. However, the endpoints of those runs might actually be distinguishable. For that reason, the correct method to use depends on the details of the list. For instance, if the "same" numbers tend to be clustered together, with different clusters being far away from each other (far meaning outside the precision, so to speak), then the Split method is a good one. If instead the list is sort of continuous-ish, then this method breaks down.

### Updated

Conceptually, I believe that Jens's second answer is the most correct of all the solutions. It is also the fastest. That is a win.

For larger lists, all of these versions are likely to give slightly different results, so I can't compare the lists directly. Nonetheless, let's do some (not very rigorous) benchmarking:

jens = DeleteDuplicates[#, (Abs[#1 - #2] < 10^-3 &)] &;
jens2 = DeleteDuplicates[Round[#, 9. 10^-4]] &;
bills = DeleteDuplicates@MeanShiftFilter[#, 2, 0.001] &;
bobh = Mean /@ Gather[#, Abs[#1 - #2] < 10^-3 &] &;
march = Mean /@ Split[Sort@#, Abs[#1 - #2] < 10^-3 &] &;


Then,

list = RandomReal[{0, 0.1}, 100];
First /@ (AbsoluteTiming[#@list;] & /@ {jens, bills, bobh, march, jens2})
(* {0.006416, 0.000253, 0.004362, 0.000271, 0.000033} *)

list = RandomReal[{0, 1}, 1000];
First /@ (AbsoluteTiming[#@list;] & /@ {jens, bills, bobh, march, jens2})
(* {0.620162, 0.000364, 0.426561, 0.002782, 0.000082} *)


(Didn't include the others because they were taking some time.)

list = RandomReal[{0, 10}, 10000];
First /@ (AbsoluteTiming[#@list;] & /@ {bills, march, jens2})
(* {0.001579, 0.027683, 0.000684} *)

list = RandomReal[{0, 100}, 100000];
First /@ (AbsoluteTiming[#@list;] & /@ {bills, march, jens2})
(* {0.015080, 0.277588, 0.008687} *)


Of course, one wonders about how the different versions scale with the density, i.e. with how many distinct elements there actually are.

list = {-0.0223, -0.0199, -0.0195, -0.0159, -0.0159, -0.0122, -0.0126};

Mean /@ Gather[list, Abs[#1 - #2] < 10^-3 &]

(*  {-0.0223, -0.0197, -0.0159, -0.0124}  *)