I want to use mathematica to symbolically solve the minimization problem in simple linear regression:

$${\text{Find }}\text{arg}\min _{\alpha ,\,\beta }Q(\alpha ,\beta ),\qquad {\text{for }}Q(\alpha ,\beta ) =\sum _{i=1}^{n}(y_{i}-\alpha -\beta x_{i})^{2},$$

where $y_i$, $x_i$, and $n$ are symbolic but not specific numbers.The expected answer would be something like

$$\begin{align} \hat {\beta }&={\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})(y_{i}-{\bar {y}})}{\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}}},\\ {\hat {\alpha }}&={\bar {y}}-{\hat {\beta }}\,{\bar {x}} \end{align}$$

Can someone show some code example for doing this? Thanks in advance!

  • $\begingroup$ Thanks Michael for your suggestions! $\endgroup$ – CrossD Jul 9 '16 at 21:49
  • $\begingroup$ See (107426). $\endgroup$ – MarcoB Jul 9 '16 at 21:54
  • $\begingroup$ What have you done so far? People are more likely to help if you show effort. $\endgroup$ – Feyre Jul 9 '16 at 21:56
  • 1
    $\begingroup$ You're welcome. Maybe someone else can help with your question. My experience has been that Mathematica's facility with algebraic/symbolic sums is limited. $\endgroup$ – Michael E2 Jul 9 '16 at 21:56

The following is an adaptation of my answer to this question, which focused on the 3D linear least-squares problem.

(* Rules to get constants out of sums (or integrals etc) *)
outrules = {
   Sum[f_ + g_, it : {x_Symbol, __}] :> Sum[f, it] + Sum[g, it],
   Sum[c_ f_, it : {x_Symbol, __}] :> c Sum[f, it] /; FreeQ[c, x],
   Sum[c_, it : {x_Symbol, __}] :> c Sum[1, it] /; FreeQ[c, x]

(*Generate the sum of squares*)
Sum[Expand[(y[i] - a x[i] - b)^2], {i, 1, n}];

(*Calculate the derivatives*)
Grad[%, {a, b}];

(*Use the linearity property of sums*)
Distribute /@ %;

(*Pull out any constants from summations*)
% //. outrules;

(*Set the derivatives equal to zero to generate a system of equations*)
Simplify[Thread[% == 0]];

(*Solve for the a, b parameters*)
Solve[%, {a, b}] // FullSimplify

Mathematica graphics

These sums are not yet expressed as a function of $(\bar{x},\bar{y})$, i.e. the average values of the $(x_i,y_i)$, respectively, as you have them in your question, but that should be a question of some algebraic transformations. Some such transformations are reported e.g. in this MathWorld document of least-squares fitting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.