Apply NestList to Iteratively Calculate Gradient (Gradient Descent)

Question

I'm attempting to calculate the gradient of a function defined by two variables. After following a tutorial I found, I have the following code:

f[x_, y_] = (2 x^4 - x^2 + 3 x - 7 y + 1)/(e^(2 x^2/3 + 3 y^2/4));
(*t is gradient of f*)
t[{x_, y_}] := {
D[f[x, y], x],
D[f[x, y], y]
};

(*
Trying to use NestList to basically calculate gradient descent 'n' times/
use Euler's method starting at the point (.510, -.445)
*)

NestList[t, {.51, -.445}, n]

However, the output I'm getting is ".51" and "-.445" are not valid variables. I understand this is because I when I apply NestList, I am essentially taking the derivative with respect to ".51" and "-.445" which isn't possible obviously. However, I am unsure of how to apply NestList with respect to the variables x and y at the point (.51,-.445).

Answer 1

What you have is not quite the way to do steepest descent. Recall that you need to pick a step size for the direction where the next point is headed. In fancy code, that is done through some form of line search, but to keep things simple here, let us use a fixed step size.

γ = 1/10; (* step size *)
iter[{x_, y_}] = {x, y} - γ D[f[x, y], {{x, y}}];

(Note that I have used here the special syntax of D[] for evaluating the gradient; of course, D[f[x, y], {{x, y}}] == {D[f[x, y], x], D[f[x, y], y]}.)

Generate the iterates until convergence and look at the last two:

its = FixedPointList[iter, {-0.8, 0.4}];
Take[its, -2]
   {{-0.414549, 0.791434}, {-0.414549, 0.791434}}

(If you only want a certain number of iterates, replace FixedPointList[] with NestList[].)

Visualize the steepest-descent path:

ContourPlot[f[x, y], {x, -1, 0}, {y, 0, 1}, 
            Epilog -> {Directive[Orange, Arrowheads[Small]], Arrow[its]}, 
            PlotRange -> All]

Answer 2

Via @kglr:

ClearAll[t];
t[{x_, y_}] = {
D[f[x, y], x], 
D[f[x, y], y]
};
NestList[t, {.51, -.445}, n]