Behavior of D and Derivative when acting on functions with Rules as arguments

Question

I was (pleasantly) surprised that when a Derivative is taken of a symbolic function for which the last few arguments are Rules, a Derivative object is returned with just the right number of arguments to ignore the Rules:

D[f[x,y,a->b],x] // InputForm
(*  Derivative[0,1][f][x,y,a->b]  *)

(*  NOT:  Derivative[0,1,0][f][x,y,a->b] *)

This makes sense since the Rules could represents options to the function f.

Question: Has this behavior of D and Derivative been with Mathematica since v1?

But, then there is a strange behavior when D is taken of a function in which a Rule appears in a middle argument:

D[f[x,a->b,y],x]
(*  (0 -> 0) Derivative[0, 1, 0][f][x, a -> b, y] + Derivative[1, 0, 0][f][x, a -> b, y] *)

D[f[x,a->b,y],y]
(*  Derivative[0, 0, 1][f][x, a -> b, y] + (0 -> 0)*Derivative[0, 1, 0][f][x, a -> b, y] *)

Question: what is the interpretation of this output? In particular, what is the meaning of the term proportional to 0 -> 0?

Answer 1

First of all, D Threads over Rule:

D[x -> x, x]
(* 1 -> 1 *)

Thus, the behavior you're seeing is just the chain rule. D sees f[x, Rule[a, b], y] as a function of three inputs, and so takes the derivative of the outside function multiplied by the inside functions:

D[f[x, a -> b, y], x]
(* (0 -> 0) Derivative[0, 1, 0][f][x, a -> b, y] + Derivative[1, 0, 0][f][x, a -> b, y] *)

Interestingly, there's something special about the way that D treats functions with Rules as arguments. Aside from the observation made in the OP (that D sees f[x, a -> b] as a function of one argument x, with the second argument understood to be Options for f), there is also the following. Even if I try to UpSet automatic Threading for D of some undefined symbol g,

g /: D[g[a_, b_], x_] := g[D[a, x], D[b, x]]
SetAttributes[g, SequenceHold]  (* to mimic the `Attributes` of `Rule` *)

the behavior noted in the OP doesn't occur:

D[f[x, g[a, b], y], x]
(* Derivative[1, 0, 0][f][x, a -> b, y] *)

which means that D looks at the structure of the expression as a whole first and decides what arguments are functions of x and which are not before applying derivatives to those arguments according to the chain rule. Of course, it does this unless one of the arguments is a Rule, in which case it does take the derivative of the argument (or it Threads over it, depending on where it is in the expression).

This is not a complete answer, since I don't know why (and exactly how) D treats Rules as arguments separately. Nonetheless, the OP is seeing a chain rule, so that's the explanation for the behavior in the OP.