1
$\begingroup$

I like to use Select statements to clean my data. Usually I will use the Select to exclude data from the next stage of my analysis/calculation

For example here is some pretend data with some outliers.

SomeFunction[A_,ω_, a0_, t_] := A Cos[ω t+t ] + a0
SomeData = 
Table[
        {
            t, (*Index or x ordinate*)
            Round[SomeFunction[1,0.123, 1.2345, t] + RandomVariate[NormalDistribution[0,0.2]],0.001], (* "measured" data *)
            Round[SomeFunction[1,0.123, 1.2345, t],0.001] (* true data, unoised*)
        },
        {t, 0,10, 0.5}
    ];

 SomeData = 
{{0., 2.452, 2.234}, {0.5, 2.529, 2.081}, {1., 1.538, 1.667}, {1.5, 
      1.334, 1.121}, {2., 0.6, 0.609}, {2.5, 0.616, 0.29}, {3., 0.149, 
      0.26}, {3.5, 0.408, 0.53}, {4., 1.141, 1.016}, {4.5, 1.696, 
      1.569}, {5., 1.833, 2.019}, {5.5, 2.574, 2.229}, {6., 2.1, 
      2.133}, {6.5, 1.773, 1.761}, {7., 1.495, 1.227}, {7.5, 0.889, 
  0.696}, {8., 0.786, 0.33}, {8.5, 0.306, 0.242}, {9., 0.313, 
  0.458}, {9.5, 1.134, 0.913}, {10., 1.422, 1.467}}

The three columns are index, the "measured" data, and the expected data

To filter these out is trivial. One can do something like: Select[SomeData, #[[2]] - #[[3]] < 0.2 &][[1;;,{1,2}]] So where the "measured" or noised data differs from the "true" or noised value by some defined threshold, the rows are excluded.

However, I instead want to replace the outlier values with some other value. The reason being that I want to preserve the structure of my data set, and by replacing the outlier points with some ludicrous value, I can filter these out right at the end of my evaluation.

The above Select statement filters rows with the index t= 2.5 , 5.5, 7, 8, 9.5. I would want to instead replace the value with 999999 for example so the output would be

{{1.`, 1.538`}, {2.`, 0.6`}, {2.5`, 999999`}, {3.`, 0.149`}, {3.5`, 
  0.408`}, {4.`, 1.141`}, {4.5`, 1.696`}, {5.`, 1.833`}, {5.5`, 
  999999`}, {6.`, 2.1`}, {6.5`, 1.773`}, {7`, 999999`}, {7.5`, 
  0.889`}, {8`, 999999`}, {8.5`, 0.306`}, {9.`, 0.313`}, {9.5`, 
  999999`}, {10.`, 1.422`}}
$\endgroup$
3
  • 2
    $\begingroup$ This doesn't seem to match your desired output, but it matches your description I think? Does SomeData /. {x : {a_, b_, c_} /; b - c < 0.2 :> Most@x, x : {a_, b_, c_} /; b - c >= 0.2 :> {a, 999999}} give you what you want? $\endgroup$ – march Feb 6 '20 at 0:08
  • $\begingroup$ Your rule replacement is elegant. I was trying to figure out how to get that to work with ReplacePart and was stumped. $\endgroup$ – Mark R Feb 6 '20 at 0:28
  • $\begingroup$ @march you should also post this as an answer $\endgroup$ – Q.P. Feb 6 '20 at 19:44
3
$\begingroup$
ClearAll[clip];
clip[δ_, repl_: 9999][{a_, b_, c_}] := 
    {a, Clip[b, {-∞, c + δ - 2 $MachineEpsilon}, {-∞, repl}]};

Map[clip[.2]] @ SomeData

{{0., 9999}, {0.5, 9999}, {1., 1.538}, {1.5, 9999}, {2., 0.6}, {2.5, 9999}, {3., 0.149}, {3.5, 0.408}, {4., 1.141}, {4.5, 1.696}, {5., 1.833}, {5.5, 9999}, {6., 2.1}, {6.5, 1.773}, {7., 9999}, {7.5, 0.889}, {8., 9999}, {8.5, 0.306}, {9., 0.313}, {9.5, 9999}, {10., 1.422}}

 Map[clip[.1, Null]] @ SomeData

{{0., Null}, {0.5, Null}, {1., 1.538}, {1.5, Null}, {2., 0.6}, {2.5, Null}, {3., 0.149}, {3.5, 0.408}, {4., Null}, {4.5, Null}, {5., 1.833}, {5.5, Null}, {6., 2.1}, {6.5, 1.773}, {7., Null}, {7.5, Null}, {8., Null}, {8.5, 0.306}, {9., 0.313}, {9.5, Null}, {10., 1.422}}

$\endgroup$
2
$\begingroup$

Would something like this do what you want:

Block[{outlierPositions = 
   Flatten[Position[SomeData, #] & /@ 
     Select[SomeData, ! (#[[2]] - #[[3]] < 0.2) &], 1], 
  replacements},
 replacements = Append[#, 2] & /@ outlierPositions;
 ReplacePart[SomeData, replacements -> 999999][[;; , {1, 2}]]
 ]

(*{{0., 999999}, {0.5, 999999}, {1., 1.538}, {1.5, 999999}, {2., 
  0.6}, {2.5, 999999}, {3., 0.149}, {3.5, 0.408}, {4., 1.141}, {4.5, 
  1.696}, {5., 1.833}, {5.5, 999999}, {6., 2.1}, {6.5, 1.773}, {7., 
  999999}, {7.5, 0.889}, {8., 999999}, {8.5, 0.306}, {9., 
  0.313}, {9.5, 999999}, {10., 1.422}}*)

This code inverts your "good criteria" to find outliers and then uses ReplacePart to do what you wanted.

$\endgroup$
1
  • $\begingroup$ This is a nice solution, +1. I will accept kglr's answer purely on the basis that it is less lines of code. What I do like about yours is that it is more transparent as to to what is going on. $\endgroup$ – Q.P. Feb 6 '20 at 19:46
1
$\begingroup$

Using replacement rules:

SomeData /. x : {a_, b_, c_} :> If[b - c < 0.2, Most@x, {a, 999999}]
(* {{0., 999999}, {0.5, 999999}, {1., 1.538}, {1.5, 999999},
    {2., 0.6}, {2.5, 999999}, {3., 0.149}, {3.5, 0.408},
    {4., 1.141}, {4.5, 1.696}, {5., 1.833}, {5.5, 999999},
    {6., 2.1}, {6.5, 1.773}, {7., 999999}, {7.5, 0.889},
    {8., 999999}, {8.5, 0.306}, {9., 0.313}, {9.5, 999999}, {10., 1.422}} *)
$\endgroup$
1
  • $\begingroup$ Thanks, although I have already accepted an answer I thought your solution was neat and deserved at least an upvote! $\endgroup$ – Q.P. Feb 6 '20 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.