2
$\begingroup$

so I have a table which comprises three columns, each column is independent measurements of the same thing and each should have some (hopefully) Gaussian error in it. I'm using the first column as basically a test to throw out outliers, so what I want to do is write a function that checks each entry in $M_{i,1}$ and if $\frac{|M_{i,1}-Mean[M_{i,1}]|}{\sqrt{Variance[M_{i,1}]}} \geq 1.5$ then I want to throw out that row.

I know how to do this procedurally but would like to understand how to do something like this functionally. Could anyone shed some light?


Edit:

Here's a sample of my data, as you can see here, I already subtracted off the mean of the first row and normalized it by the error, so now the question is just how to I delete the rows where the first value is > 1.5?

data = Import["/home/pi/Documents/Wolfram Mathematica/OT/run.1.dat"];
m = data[[1 ;; 10]];
m[[All, 1]] = 
  1.5*(m[[All, 1]] - Mean[m[[All, 1]]])/Sqrt[Variance[m[[All, 1]]]];
m

which gives

{{1.79446, 243.9, 236.019}, 
{1.25795, 245.783, 234.851}, 
{-2.74877, 239.893, 235.587}, 
{-0.305926, 242.533, 236.141}, 
{1.39874, 241.313, 237.124}, 
{-0.754923, 242.461, 235.76}, 
{-0.370612, 241.358, 236.501}, 
{1.67651, 241.142, 237.51}, 
{-0.5114, 243.559, 236.405}, 
{-1.43603, 242.929, 235.823}}

And I now want to delete all the rows where Abs[the first value]>1.5, hence I want it to look like

{1.25795, 245.783, 234.851}, 
{-0.305926, 242.533, 236.141}, 
{1.39874, 241.313, 237.124}, 
{-0.754923, 242.461, 235.76}, 
{-0.370612, 241.358, 236.501}, 
{-0.5114, 243.559, 236.405}, 
{-1.43603, 242.929, 235.823}}
$\endgroup$
2
  • 1
    $\begingroup$ Please edit your post to provide a sample data set (not too long!) with the desired output for that dataset. In any cases, look up Select or DeleteCases. $\endgroup$
    – march
    Commented Oct 18, 2016 at 22:27
  • $\begingroup$ If you want to remove outliers, should you perhaps have an absolute value on the LHS of your inequality...? $\endgroup$ Commented Oct 18, 2016 at 22:43

1 Answer 1

1
$\begingroup$

I would use Select to keep the wanted ones:

With[{
  mean = Mean @ m[[All, 1]],
  sigma = Sqrt @ Variance @ m[[All, 1]]
},
Select[m, (First @ # - mean)/sigma < 1.5 & ]
]

where m is your table should work (avoid using uppercase letters as variable names).

If the number of rows is very large then the following may be a bit faster:

Pick[m, UnitStep[(m[[All, 1]] - Mean @ m[[All, 1]])/Sqrt[Variance @ m[[All, 1]]] - 1.5], 0]
$\endgroup$
6
  • $\begingroup$ Crap! It didn't actually work, I tried plotting the normalized residuals from the truncated data set and the outliers are still all there. In fact, I can't really tell what was actually truncated out of it... $\endgroup$
    – Mason
    Commented Oct 19, 2016 at 17:15
  • $\begingroup$ As I wrote in a comment to the question, if you want to remove outliers then shouldn't you have an absolute value on the LHS of the inequality? Try modifying to account for that. Also, it would help to post a small example data set ;) $\endgroup$ Commented Oct 19, 2016 at 19:55
  • $\begingroup$ I did use the absolute values, will post sample data soon $\endgroup$
    – Mason
    Commented Oct 20, 2016 at 1:48
  • $\begingroup$ Okay, I've updated the question. Thanks again for the help $\endgroup$
    – Mason
    Commented Oct 20, 2016 at 3:56
  • $\begingroup$ With the modified m you have posted, Select[m, Abs[First@#] < 1.5 &] works fine... But why do you multiply by 1.5, and then compare the Abs to 1.5 again...? That's not what the equation in your original question does... $\endgroup$ Commented Oct 20, 2016 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.