# Is there any automatic differentiation package?

I'm wondering if an automatic differentiation package exists for Mathematica. This is what I mean by automatic differentiation.

• I don't know; but it's interesting to note that in Mathematica the distinction between a program and a symbolic expression is blurred. Many of the simple cases will just work with D/Derivative. It will be interesting to see some rudimentary implementations. See also forums.wolfram.com/mathgroup/archive/2008/Feb/msg00381.html Feb 16, 2012 at 15:19
• To add to what Szabolcs said, for pure Mathematica expressions, symbolic differentiation is pretty much a generalization of automatic differentiation. I would guess that for these expressions, true automatic differentiation may not be any more efficient than using D. Feb 16, 2012 at 18:12
• In the link you provided are listed different packages, some of which are written in languages that can communicate with Mathematica. See demonstrations.wolfram.com/AutomaticDifferentiation and wolfram.com/products/applications/acegen
– CHM
Feb 16, 2012 at 23:41
• This is a very relevant question. It is not fun to give optimization routines like FindMinimum huge expressions generated by running some algorithm so that it can do symbolic differentiation on them. Falling back to numerical finite differences seems like a waste compared to the automatic differentation idea (see e.g. justindomke.wordpress.com/2009/02/17/… ). Mar 10, 2017 at 23:36

For anyone still interested in this: I'm working on an implementation of dual numbers for Mathematica which should allow you to calculate automatic derivatives of (hopefully) many programs.

Check it out on GitHub.

Here's a quick example of how to obtain the derivative of a programmatic function with a While loop (adapted from the documentation). It's a fixed point algorithm to solve an equation:

f[a_?NumericQ, x0 : _?NumericQ : 1.0] :=
Module[{ x = x0, y, i = 0},
While[(y = Cos[a x]) != x,
x = y;
i++
];
x
];


It doesn't have a symbolic derivative:

Derivative[1] @ f
(* Derivative[1][f] *)


You can find the derivative of its first argument simply by passing a Dual with non-standard part equal to 1:

<<DualNumbers
f[1.]
f[Dual[1., 1.]]
(* 0.739085 *)
(* Dual[0.739085, -0.297474] *)


The first argument of Dual is the function value and the second argument is the derivative at that point. Let's convince ourselves that this derivative is correct:

With[{h = 0.001, a = 1.0},
(f[a + h] - f[a - h])/(2 h)
]
(* -0.297474 *)


Let's also check the derivative of the second argument:

f[1., Dual[1., 1.]]
(* Dual[0.739085, 1.86543*10^-14] *)
`

The derivative of the second argument is pretty much zero. This makes sense, of course, since small variations in the initial value in the fixed point search shouldn't influence the result.

• I'm intrigued! Could you add an example or two here of using it for automatic differentiation? Sep 14, 2020 at 17:38
• @ChrisK I'm still working on documentation and examples, but this should give you an idea. If you have any ideas to try it out on, please let me know :). Sep 14, 2020 at 18:20
• For neural networks, I always wanted something that could derive backward AD in matrix or einsum notation. For instance results in people.maths.ox.ac.uk/gilesm/files/NA-08-01.pdf . When I was on tensorflow team, we derived them by hand, basically through "guess-and-check" Sep 14, 2020 at 19:00
• @YaroslavBulatov That's a good idea, but I've not used that package before so it might be a while for me to look into this. Feel free to make a pull request to my repo if you have any ideas about this. Sep 14, 2020 at 19:15
• For anyone interested in the question by @YaroslavBulatov : see the discussion on GitHub: github.com/ssmit1986/DualNumbers/issues/1 Sep 16, 2020 at 6:30

Try this. It may not be exactly what your looking for, but it also may give you a good starting point.