# Derivative of rectified linear unit in Mathematica?

I am following the artificial neural networks literature and apparently the latest trend is to use the rectified linear units (ReLU) as the activation functions for each neuron. I tried to take the derivative of this function in Mathematica but it gives indeterminate at x=0:

f[x_]:= Max[0,x]
D[f[x],x]


In the neural network computation, one explicitly defines the derivative of the ReLU at x=0 as 0. Can we also instruct Mathematica to do the same? Do we have to define the derivative at x=0 explicitly somehow? Or there is another trick here? Since Mathematica 11 now has the deep learning tools, I am assuming that this problem must have been addressed there?

Thanks.

You can use

f = Ramp
Derivative[1][f] = x \[Function] Piecewise[{{1, x > 0}}]

• or Derivative[1][f] = f' /. Indeterminate -> 0. – AccidentalFourierTransform May 2 '18 at 19:27
• Or that. But that would unpack arrays. – Henrik Schumacher May 2 '18 at 19:28
• Hmm good point. – AccidentalFourierTransform May 2 '18 at 19:29
• Oh sorry, just realized: It does't unpack. – Henrik Schumacher May 2 '18 at 20:49

You can explicitly define the function f using Piecewise (taking care of the endpoints)

f[x_] := Piecewise[{{x, x > 0}, {0, x <= 0}}];


Then the derivative of f is itself the Piecewise function

Dt[f[x], x]


A little more playing around shows that you get the same thing using the Max function:

f[x_] := Max[0, x]
Dt[f[x], x]

• Hm. I am not sure, but I think the numerical optimization algorithms such as FindMinimum would apply D and not Dt so that bad thinks might happen. For example, f' still contains Indeterminate. (I wonder why Dt doesn't show a Indeterminate for x == 0...) – Henrik Schumacher May 2 '18 at 19:43