I am following the online Stanford Course in Machine Learning and I am trying to implement the algorithms on the fly in Mathematica.
Currently, I have issues with minimizing the Locally Weighted Linear Regression -- it hangs and just runs for more than 3 hours. I was hoping you might have some clues as to why.
The code I have is the following
trainX = {100, 320, 213, 512, 58, 84, 113, 142, 93, 121, 421, 432,
249, 254};
trainY = {140000, 400000, 241000, 489000, 78000, 123000, 139000,
143000, 97000, 134000, 392000, 458000, 311000, 378000};
trainComposed =
Transpose@{trainY, Table[1, {Length[trainX]}], trainX};
I define my hypothesis as
Hyp[x_, T_] := T.x;
and the cost function as
Cost[T_, case_, t_] :=
1/2 Total[Exp[-(({##2} - case).({##2} - case)/(2t^2))]
(Hyp[{##2}, T] - #1)^2 & @@@ trainComposed]
where T
is a list of the parameters we are trying to find, case
is the particular values on x
for which we want to fit the model, and t
is an arbitrary parameter that defines the weight of neighbouring data points.
I can calculate the cost without any issues, but when I try to minimise, it hangs. The code I use to run it with is
Clear[v]
T = Array[v, Length[trainComposed[[1]]] - 1];
Minimize[Cost[T, {1, 121}, 50], T]
I am essentially trying to minimise the following function
$$\sum_iw^{(i)}(y^{(i)}-\theta^Tx^{(i)})^2$$ where $$w^{(i)}=\exp\left(-\frac{(x^{(i)}-x)^2}{2\tau^2}\right)$$
and $y^{(i)}$ corresponds to the target value for the $y^{th}$ training example, $x^{(i)}$ is a feature vector of the $i^\text{th}$ example, and $x$ is the position vector for which we are currently doing the fit.
What am I doing wrong?
Minimize[]
? Do you really need symbolic results? If not, useNMinimize[]
instead. $\endgroup$NMinimize
gives immediate results. $\endgroup$