I am following the artificial neural networks literature and apparently the latest trend is to use the rectified linear units (ReLU) as the activation functions for each neuron. I tried to take the derivative of this function in Mathematica but it gives indeterminate at x=0:

f[x_]:= Max[0,x]

In the neural network computation, one explicitly defines the derivative of the ReLU at x=0 as 0. Can we also instruct Mathematica to do the same? Do we have to define the derivative at x=0 explicitly somehow? Or there is another trick here? Since Mathematica 11 now has the deep learning tools, I am assuming that this problem must have been addressed there?



You can use

f = Ramp
Derivative[1][f] = x \[Function] Piecewise[{{1, x > 0}}]
  • 1
    $\begingroup$ or Derivative[1][f] = f' /. Indeterminate -> 0. $\endgroup$ – AccidentalFourierTransform May 2 '18 at 19:27
  • $\begingroup$ Or that. But that would unpack arrays. $\endgroup$ – Henrik Schumacher May 2 '18 at 19:28
  • $\begingroup$ Hmm good point. $\endgroup$ – AccidentalFourierTransform May 2 '18 at 19:29
  • $\begingroup$ Oh sorry, just realized: It does't unpack. $\endgroup$ – Henrik Schumacher May 2 '18 at 20:49

You can explicitly define the function f using Piecewise (taking care of the endpoints)

f[x_] := Piecewise[{{x, x > 0}, {0, x <= 0}}];

Then the derivative of f is itself the Piecewise function

Dt[f[x], x]

enter image description here

A little more playing around shows that you get the same thing using the Max function:

f[x_] := Max[0, x]
Dt[f[x], x]
  • 1
    $\begingroup$ Hm. I am not sure, but I think the numerical optimization algorithms such as FindMinimum would apply D and not Dt so that bad thinks might happen. For example, f' still contains Indeterminate. (I wonder why Dt doesn't show a Indeterminate for x == 0...) $\endgroup$ – Henrik Schumacher May 2 '18 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.