1
$\begingroup$

Imagine I have data points in triplicates, as follows $$ L=\begin{pmatrix} p_{11} & p_{12} & p_{13}\\ p_{21} & p_{22} & p_{23}\\ & \vdots &\\ p_{n1} & p_{n2} & p_{n3} \end{pmatrix} $$ What is the best way to filter this data, so that I remove outliers and end up with a vector of best-fit values $$ \tilde{L}=(\tilde{p}_1,\tilde{p}_2,\cdots,\tilde{p}_n) $$ Thoughts: For example, if I have the plot

list = Table[
   If[j == 5, {0.6, 0.7, 0.1}, 
    RandomReal[{0. + 0.1 j, 0.3 + 0.1 j}, 3]], {j, 6}];
ListPlot[Transpose@list, PlotStyle -> Black, Frame -> True, 
 PlotRange -> All]

enter image description here

I could naturally take the mean of these values (in blue)

Show[
 ListPlot[Transpose@list, PlotStyle -> Black, Frame -> True, 
  PlotRange -> All],
 ListPlot[Transpose@{Range[6], Mean /@ list}, PlotStyle -> Blue]
 ]

enter image description here

but clearly, at $n=5$ I have a potential outlier, which could be removed for a better fit. What would be the ideal technique to assign a weighted mean to each triplicate based on the overall dataset, possibly based on the average standard deviation between triplicates? I have heard about Cook's distance, but I am not too sure how to apply it here.

Any ideas?

$\endgroup$
1
  • 4
    $\begingroup$ This is a good question but it would seem to be a better question for stats.stackexchange.com. And there you'll certainly be asked about why you're tossing out data. For example, is there some reason for tossing a point (such as the thermocouple broke) rather than the point just doesn't look like it belongs. $\endgroup$
    – JimB
    Commented Feb 23 at 16:36

1 Answer 1

1
$\begingroup$

To eliminate outlier, you may e.g. fit a linear function to your data and subtract it from the data to get the deviations. Then calculate the standard deviation: std of the deviations. Now you can reject data points that differ more than some multiple (here e.g. 2) of the std from the fit. Here is an example:

dat = Flatten[
   Table[{j, 
     If[j == 5, {0.6, 0.7, 0.1}[[i]], 
      RandomReal[{0. + 0.1  j, 0.3 + 0.1  j}]]}, {j, 6}, {i, 3}], 1];
lin[x_] = Fit[dat, {1, x}, x];
std = StandardDeviation[t = dat[[All, 2]] - lin /@ dat[[All, 1]]];
dat1 = DeleteCases[dat, {x_, y_} /; Abs[y - lin[x]] > 2  std];
ListPlot[dat1, ImageSize -> {300, 200}]

enter image description here

$\endgroup$
2
  • $\begingroup$ Sorry, what do you mean by "Then calculate the standard deviation: std of the deviations"? $\endgroup$
    – sam wolfe
    Commented Apr 15 at 15:48
  • $\begingroup$ The deviations are the difference between a data point and the corresponding point on the linear fit. The deviations have a mean and a standard deviation: std. $\endgroup$ Commented Apr 15 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.