I'm sure this has been answered before, but I haven't been able to find it. I'm trying to manually implement gradient descent for some function, f, with a vector of any length.

I define the update rule as follows for a vector with two entries:

a = {0.1, 0.01};
Step[th_] := th + a Grad[f[{x, y}], {x, y}] /. {x -> th[[1]], y ->th[[2]]}

My question then becomes, how would I do this for a vector of n-dimensions without having to manually key in all the replacements? Ideally, I would just check the dimension of my input parameter, and then follow through with the correct calculations. How can I achieve that?

  • $\begingroup$ Ponder on the following snippet: With[{vec = {u, v, w}}, vec + (a (Derivative[##][f] @@ vec) & @@@ IdentityMatrix[Length[vec]])] $\endgroup$ Jun 15, 2016 at 12:36
  • $\begingroup$ Thanks @J.M. It didn't evaluate completely though. I had to change it to the following: Step[th] := With[{x = th}, x + a ((Derivative[##][l] @@ {x}) & /@ IdentityMatrix[Length[x]])]. For some reason, the apply function didn't evaluate my function, and I had to change it to the map. $\endgroup$ Jun 15, 2016 at 13:52
  • $\begingroup$ Yes, your modifications are the right one for the case where the argument is a $k$-vector instead of a function with $k$ arguments. $\endgroup$ Jun 15, 2016 at 16:15


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.