Skip to main content
4 of 10
deleted 55 characters in body
partida
  • 6.9k
  • 24
  • 48

How to make Advanced Activations Layers in Mathematica?

I want to try LeakyReLU Layer in Mathematica,but not found.

In keras,the LeakyReLU be defined like this:

enter image description here

So I try to use ElementwiseLayer to implement it.

g = Function[x, Piecewise[{{0.3(*default value*)*x, x < 0}, {x, x >= 0}}], Listable];
Plot[g[x], {x, -2, 2}]
g[{-1, -0.5, 0, 0.5, 1}]

enter image description here

But ElementwiseLayer[g] failed.

Then I try:

f = HeavisideTheta[#]*# + HeavisideTheta[-#]*0.3# &;
f[{-1, -0.5, 0, 0.5, 1}]

But ElementwiseLayer\[f\] also failed.

enter image description here

It also failed,so it's confused me, how to make LeakyReLU or other advanced activations layers in Mathematica?

partida
  • 6.9k
  • 24
  • 48