3
$\begingroup$

I have data which looks something like this:

invalid points

And I want to approximate it with a line

The points below the line are outliers, and I want to ignore them when estimating a and b in a*x + b

What is the best way to accomplish this in mathematica?

Here's the sample data:

{{0, -1.46871}, {1, -0.960396}, {2, -0.933101}, {3, -0.193156}, {4, 
-0.787081}, {5, 0.349225}, {6, -1.04288}, {7, -0.641152}, {8, 
  0.44213}, {9, 0.335079}, {10, 1.62793}, {11, 2.09465}, {12, 
  0.692115}, {13, 2.58813}, {14, 2.06381}, {15, 3.35164}, {16, 
  2.71424}, {17, 2.81021}, {18, 3.09869}, {19, 3.54265}, {20, 
  4.84738}, {0., -9.16012}, {5.09981, -8.19077}, {10.1996, -5.36297}, 
{15.2994, -3.10673}}
$\endgroup$
  • $\begingroup$ You can select all of the points whose ordinate is greater than -2 by evaluating Select[data, #[[2]] > -2 &], where data is your list of points. $\endgroup$ – LouisB May 17 '17 at 7:26
  • $\begingroup$ Yes, but I want to automate the process. And the data set and outliers are not guaranteed to have a horizontal divider $\endgroup$ – Arsen Zahray May 17 '17 at 8:15
  • 1
    $\begingroup$ Possible duplicate of "Filtering and Replacing outliers" and "How to remove outliers from data." $\endgroup$ – Alexey Popkov May 17 '17 at 10:12
  • $\begingroup$ "And the data set and outliers are not guaranteed to have a horizontal divider"...You will have to decide on a criteria to eliminate the points. $\endgroup$ – Lotus May 17 '17 at 11:27
3
$\begingroup$

You can use FindClustersand select the required data.

dataFilt = FindClusters[data] // First;
lm = LinearModelFit[dataFilt, x, x]

enter image description here

Show[ListPlot[data], Plot[lm[x], {x, 0, 20}], Frame -> True]

enter image description here

$\endgroup$
  • $\begingroup$ Tried FindClusters. It finds only 1 cluster on the example data, and that does not exclude the points in question $\endgroup$ – Arsen Zahray May 17 '17 at 7:55
  • $\begingroup$ @ArsenZahray I'm on version 11.1.0. I just checked it on 10.4.1 and it returns a single cluster. Looks like they have improved FindClusters[] function in the recent releases. $\endgroup$ – Anjan Kumar May 17 '17 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.