# How many different methods of least square estimation can be performed in mathematica?

Let's take a simple case
$$y_i = \beta_0 + \beta_1 x_i$$ $$x_i \,\,\,\, y_i$$ are the observed data of independent variable and the dependent variable respectively. $$\beta_0 \,\,\,\,\beta_1$$ are estimated by minimizing the sum of vertical distances of the observed data from the fitted line
$$s\left(\beta_0, \beta_1\right) = \Sigma_i (\epsilon_i)^2 = \Sigma_i (y_i - \beta_0 - \beta_1 x_i)^2$$ this is known as ordinary least square estimate or direct regression.
The other methods of parameter estimation consists of
(i) Inverse regression: sum of horizontal distances of the observed data from the fitted line is minimized
(ii) Major axis regression: sum of perpendicular distances of the observed data from the fitted line is minimized
http://mathworld.wolfram.com/LeastSquaresFittingPerpendicularOffsets.html
(iii) Reduced major axis regression: the sum of the areas of rectangles, defined between the observed data points and the nearest point on the fitted line, is minimized.
etc..
For more details please refer to http://home.iitk.ac.in/~shalab/regression/Chapter2-Regression-SimpleLinearRegressionAnalysis.pdf page 2-4.
LinearModelFit may use ordinary least square estimate by default. Can the other three methods be performed in mathematica? If so how?

• Of course you can develop code to implement any method you wish – george2079 Apr 16 '18 at 11:31
• Is there anything readily available in mathematica? – csk Apr 16 '18 at 11:33
• Not readily available in the sense of a built in function . 2 is trivially done by reversing the data. 3 needs nminimize. – george2079 Apr 16 '18 at 11:45
• ok let me have a look at nminimize. what about Reduced major axis regression, the last one I mentioned? – csk Apr 16 '18 at 11:51
• This is definitely more information than provided in your previous question: mathematica.stackexchange.com/questions/170477/…. But next time you might want to edit a closed question rather than opening a new one with the same question. – JimB Apr 16 '18 at 16:00

Reduced major axis regression:

Example data:

SeedRandom[0]
data = Table[  {# + RandomVariate[NormalDistribution[0, .1]],
1 + # + RandomVariate[NormalDistribution[0, .1]]} &@
RandomReal[{0, 1}] , {12}] ;


standard linear regression for comparison:

fit = LinearModelFit[data, x, x][x]


error defined as (positive) area of rectangle between point and nearest point to line:

errRMA[m_?NumericQ, b_?NumericQ, pt_List] :=
Abs[Times @@ Subtract @@@
Transpose[{({{1, m}, {m, m^2}}.pt - {m, -1} b)/(1 + m^2), pt}]]


note the distance formula can be found from RegionNearest[InfiniteLine[{{0,b},{1,m+b}}],{p,q}]

minimize error

rmafit = m x + b /.
FindMinimum[Total[errRMA[m, b  , #] & /@ data  ] , {m, b}][[2]]
Show[{ListPlot[data, AspectRatio -> 1],
Plot[{fit, rmafit }, {x, 0, 1},
PlotStyle -> {Dashed, Black}]}]


illustration:

nr = RegionNearest[Line[{{0, rmafit /. x -> 0}, {1, rmafit /. x -> 1}}]];
Show[{
ListPlot[data,
Epilog -> ({FaceForm[None], EdgeForm[Dashed],
Rectangle[#, nr[#]] & /@ data})],
Plot[rmafit , {x, 0, 1}, PlotStyle -> Black]}, AspectRatio -> 1]


aside, note that the reduced major axis regression error formulation has degenerate minima. If you fit a line exactly horizontal or vertical then all the rectangles have zero area, hence zero error. For robustness you probably want to do an ordinary linear regression and use those parameters as the initial guess for FindMinimum.

• Thanks. I'll look into it in detail. – csk Apr 19 '18 at 10:47