# Using select to replace values which meet a condition

I like to use Select statements to clean my data. Usually I will use the Select to exclude data from the next stage of my analysis/calculation

For example here is some pretend data with some outliers.

SomeFunction[A_,ω_, a0_, t_] := A Cos[ω t+t ] + a0
SomeData =
Table[
{
t, (*Index or x ordinate*)
Round[SomeFunction[1,0.123, 1.2345, t] + RandomVariate[NormalDistribution[0,0.2]],0.001], (* "measured" data *)
Round[SomeFunction[1,0.123, 1.2345, t],0.001] (* true data, unoised*)
},
{t, 0,10, 0.5}
];

SomeData =
{{0., 2.452, 2.234}, {0.5, 2.529, 2.081}, {1., 1.538, 1.667}, {1.5,
1.334, 1.121}, {2., 0.6, 0.609}, {2.5, 0.616, 0.29}, {3., 0.149,
0.26}, {3.5, 0.408, 0.53}, {4., 1.141, 1.016}, {4.5, 1.696,
1.569}, {5., 1.833, 2.019}, {5.5, 2.574, 2.229}, {6., 2.1,
2.133}, {6.5, 1.773, 1.761}, {7., 1.495, 1.227}, {7.5, 0.889,
0.696}, {8., 0.786, 0.33}, {8.5, 0.306, 0.242}, {9., 0.313,
0.458}, {9.5, 1.134, 0.913}, {10., 1.422, 1.467}}


The three columns are index, the "measured" data, and the expected data

To filter these out is trivial. One can do something like: Select[SomeData, #[[2]] - #[[3]] < 0.2 &][[1;;,{1,2}]] So where the "measured" or noised data differs from the "true" or noised value by some defined threshold, the rows are excluded.

However, I instead want to replace the outlier values with some other value. The reason being that I want to preserve the structure of my data set, and by replacing the outlier points with some ludicrous value, I can filter these out right at the end of my evaluation.

The above Select statement filters rows with the index t= 2.5 , 5.5, 7, 8, 9.5. I would want to instead replace the value with 999999 for example so the output would be

{{1., 1.538}, {2., 0.6}, {2.5, 999999}, {3., 0.149}, {3.5,
0.408}, {4., 1.141}, {4.5, 1.696}, {5., 1.833}, {5.5,
999999}, {6., 2.1}, {6.5, 1.773}, {7, 999999}, {7.5,
0.889}, {8, 999999}, {8.5, 0.306}, {9., 0.313}, {9.5,
999999}, {10., 1.422}}

• This doesn't seem to match your desired output, but it matches your description I think? Does SomeData /. {x : {a_, b_, c_} /; b - c < 0.2 :> Most@x, x : {a_, b_, c_} /; b - c >= 0.2 :> {a, 999999}} give you what you want? Commented Feb 6, 2020 at 0:08
• Your rule replacement is elegant. I was trying to figure out how to get that to work with ReplacePart and was stumped. Commented Feb 6, 2020 at 0:28
• @march you should also post this as an answer Commented Feb 6, 2020 at 19:44

ClearAll[clip];
clip[δ_, repl_: 9999][{a_, b_, c_}] :=
{a, Clip[b, {-∞, c + δ - 2 \$MachineEpsilon}, {-∞, repl}]};

Map[clip[.2]] @ SomeData


{{0., 9999}, {0.5, 9999}, {1., 1.538}, {1.5, 9999}, {2., 0.6}, {2.5, 9999}, {3., 0.149}, {3.5, 0.408}, {4., 1.141}, {4.5, 1.696}, {5., 1.833}, {5.5, 9999}, {6., 2.1}, {6.5, 1.773}, {7., 9999}, {7.5, 0.889}, {8., 9999}, {8.5, 0.306}, {9., 0.313}, {9.5, 9999}, {10., 1.422}}

 Map[clip[.1, Null]] @ SomeData


{{0., Null}, {0.5, Null}, {1., 1.538}, {1.5, Null}, {2., 0.6}, {2.5, Null}, {3., 0.149}, {3.5, 0.408}, {4., Null}, {4.5, Null}, {5., 1.833}, {5.5, Null}, {6., 2.1}, {6.5, 1.773}, {7., Null}, {7.5, Null}, {8., Null}, {8.5, 0.306}, {9., 0.313}, {9.5, Null}, {10., 1.422}}

Would something like this do what you want:

Block[{outlierPositions =
Flatten[Position[SomeData, #] & /@
Select[SomeData, ! (#[[2]] - #[[3]] < 0.2) &], 1],
replacements},
replacements = Append[#, 2] & /@ outlierPositions;
ReplacePart[SomeData, replacements -> 999999][[;; , {1, 2}]]
]

(*{{0., 999999}, {0.5, 999999}, {1., 1.538}, {1.5, 999999}, {2.,
0.6}, {2.5, 999999}, {3., 0.149}, {3.5, 0.408}, {4., 1.141}, {4.5,
1.696}, {5., 1.833}, {5.5, 999999}, {6., 2.1}, {6.5, 1.773}, {7.,
999999}, {7.5, 0.889}, {8., 999999}, {8.5, 0.306}, {9.,
0.313}, {9.5, 999999}, {10., 1.422}}*)


This code inverts your "good criteria" to find outliers and then uses ReplacePart to do what you wanted.

• This is a nice solution, +1. I will accept kglr's answer purely on the basis that it is less lines of code. What I do like about yours is that it is more transparent as to to what is going on. Commented Feb 6, 2020 at 19:46

Using replacement rules:

SomeData /. x : {a_, b_, c_} :> If[b - c < 0.2, Most@x, {a, 999999}]
(* {{0., 999999}, {0.5, 999999}, {1., 1.538}, {1.5, 999999},
{2., 0.6}, {2.5, 999999}, {3., 0.149}, {3.5, 0.408},
{4., 1.141}, {4.5, 1.696}, {5., 1.833}, {5.5, 999999},
{6., 2.1}, {6.5, 1.773}, {7., 999999}, {7.5, 0.889},
{8., 999999}, {8.5, 0.306}, {9., 0.313}, {9.5, 999999}, {10., 1.422}} *)

• Thanks, although I have already accepted an answer I thought your solution was neat and deserved at least an upvote! Commented Feb 6, 2020 at 19:52