I want to try LeakyReLU Layer in Mathematica
,but not found.
In keras
,the LeakyReLU
be defined like this:
So I try to use ElementwiseLayer
to implement it.
g = Function[x, Piecewise[{{0.3(*default value*)*x, x < 0}, {x, x >= 0}}], Listable];
Plot[g[x], {x, -2, 2}]
g[{-1, -0.5, 0, 0.5, 1}]
But ElementwiseLayer[g]
failed.
Then I try:
f = HeavisideTheta[#]*# + HeavisideTheta[-#]*0.3# &;
f[{-1, -0.5, 0, 0.5, 1}]
It also failed,so it's confused me, how to make LeakyReLU or other advanced activations layers in Mathematica?