Derivative of rectified linear unit in Mathematica?

Question

I am following the artificial neural networks literature and apparently the latest trend is to use the rectified linear units (ReLU) as the activation functions for each neuron. I tried to take the derivative of this function in Mathematica but it gives indeterminate at x=0:

f[x_]:= Max[0,x]
D[f[x],x]

In the neural network computation, one explicitly defines the derivative of the ReLU at x=0 as 0. Can we also instruct Mathematica to do the same? Do we have to define the derivative at x=0 explicitly somehow? Or there is another trick here? Since Mathematica 11 now has the deep learning tools, I am assuming that this problem must have been addressed there?

Thanks.

Answer 1

You can use

f = Ramp
Derivative[1][f] = x \[Function] Piecewise[{{1, x > 0}}]

Answer 2

You can explicitly define the function f using Piecewise (taking care of the endpoints)

f[x_] := Piecewise[{{x, x > 0}, {0, x <= 0}}];

Then the derivative of f is itself the Piecewise function

Dt[f[x], x]

A little more playing around shows that you get the same thing using the Max function:

f[x_] := Max[0, x]
Dt[f[x], x]