2
$\begingroup$

so I have a table which comprises three columns, each column is independent measurements of the same thing and each should have some (hopefully) Gaussian error in it. I'm using the first column as basically a test to throw out outliers, so what I want to do is write a function that checks each entry in $M_{i,1}$ and if $\frac{|M_{i,1}-Mean[M_{i,1}]|}{\sqrt{Variance[M_{i,1}]}} \geq 1.5$ then I want to throw out that row.

I know how to do this procedurally but would like to understand how to do something like this functionally. Could anyone shed some light?


Edit:

Here's a sample of my data, as you can see here, I already subtracted off the mean of the first row and normalized it by the error, so now the question is just how to I delete the rows where the first value is > 1.5?

data = Import["/home/pi/Documents/Wolfram Mathematica/OT/run.1.dat"];
m = data[[1 ;; 10]];
m[[All, 1]] = 
  1.5*(m[[All, 1]] - Mean[m[[All, 1]]])/Sqrt[Variance[m[[All, 1]]]];
m

which gives

{{1.79446, 243.9, 236.019}, 
{1.25795, 245.783, 234.851}, 
{-2.74877, 239.893, 235.587}, 
{-0.305926, 242.533, 236.141}, 
{1.39874, 241.313, 237.124}, 
{-0.754923, 242.461, 235.76}, 
{-0.370612, 241.358, 236.501}, 
{1.67651, 241.142, 237.51}, 
{-0.5114, 243.559, 236.405}, 
{-1.43603, 242.929, 235.823}}

And I now want to delete all the rows where Abs[the first value]>1.5, hence I want it to look like

{1.25795, 245.783, 234.851}, 
{-0.305926, 242.533, 236.141}, 
{1.39874, 241.313, 237.124}, 
{-0.754923, 242.461, 235.76}, 
{-0.370612, 241.358, 236.501}, 
{-0.5114, 243.559, 236.405}, 
{-1.43603, 242.929, 235.823}}
$\endgroup$
  • 1
    $\begingroup$ Please edit your post to provide a sample data set (not too long!) with the desired output for that dataset. In any cases, look up Select or DeleteCases. $\endgroup$ – march Oct 18 '16 at 22:27
  • $\begingroup$ If you want to remove outliers, should you perhaps have an absolute value on the LHS of your inequality...? $\endgroup$ – Marius Ladegård Meyer Oct 18 '16 at 22:43
1
$\begingroup$

I would use Select to keep the wanted ones:

With[{
  mean = Mean @ m[[All, 1]],
  sigma = Sqrt @ Variance @ m[[All, 1]]
},
Select[m, (First @ # - mean)/sigma < 1.5 & ]
]

where m is your table should work (avoid using uppercase letters as variable names).

If the number of rows is very large then the following may be a bit faster:

Pick[m, UnitStep[(m[[All, 1]] - Mean @ m[[All, 1]])/Sqrt[Variance @ m[[All, 1]]] - 1.5], 0]
| improve this answer | |
$\endgroup$
  • $\begingroup$ Crap! It didn't actually work, I tried plotting the normalized residuals from the truncated data set and the outliers are still all there. In fact, I can't really tell what was actually truncated out of it... $\endgroup$ – Mason Oct 19 '16 at 17:15
  • $\begingroup$ As I wrote in a comment to the question, if you want to remove outliers then shouldn't you have an absolute value on the LHS of the inequality? Try modifying to account for that. Also, it would help to post a small example data set ;) $\endgroup$ – Marius Ladegård Meyer Oct 19 '16 at 19:55
  • $\begingroup$ I did use the absolute values, will post sample data soon $\endgroup$ – Mason Oct 20 '16 at 1:48
  • $\begingroup$ Okay, I've updated the question. Thanks again for the help $\endgroup$ – Mason Oct 20 '16 at 3:56
  • $\begingroup$ With the modified m you have posted, Select[m, Abs[First@#] < 1.5 &] works fine... But why do you multiply by 1.5, and then compare the Abs to 1.5 again...? That's not what the equation in your original question does... $\endgroup$ – Marius Ladegård Meyer Oct 20 '16 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.