# How to make Advanced Activations Layers in Mathematica?

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha，but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)


Is there any more simple method to make Advanced Activations Layers?

Application: this post used leayReLU[alpha_] := ElementwiseLayer[Ramp[#] - alpha*Ramp[-#] &]

• Setting Attribute to Listable seems to be the problem. You can try with Function directly: g = Function[x, Piecewise[{{0.3*x, x < 0}, {x, x > 0}}], Listable] May 22, 2017 at 4:53
• @AnjanKumar Thank you,I edit my question. May 22, 2017 at 5:12
• Regarding the learned parameter of PReLU, I think you need to use ConstantArrayLayer for learned constants, and (I guess) ThreadingLayer to combine input from the constant and the "data" input layers May 22, 2017 at 7:09

The documentation of ElementwiseLayer explicitly lists which functions are allowed, and UnitStep is not in this list, I believe that's why the function fails.

Simple workaround: Use a combination of functions from that list, like Ramp:

f = Ramp[#] - Ramp[-#]*0.3 &;

l = ElementwiseLayer[f]
l[{-1, -0.5, 0, 0.5, 1}]


{-0.3, -0.15, 0., 0.5, 1.}

This is how to constract a PReLU

data = Thread[RandomReal[1, {100, 2}] -> RandomReal[1, {100, 3}]];
net = NetGraph[{5, ConstantArrayLayer["Output" -> 1],
ReplicateLayer[5], FlattenLayer[],

NetTrain[net, data, MaxTrainingRounds -> 100, BatchSize -> 32];

In 2020, the function ParametricRampLayer was introduced to implement leaky ReLU layers. The slope can be either hard-coded or learned.