12
$\begingroup$

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha，but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

Application: this post used leayReLU[alpha_] := ElementwiseLayer[Ramp[#] - alpha*Ramp[-#] &]

Improve this question
$\endgroup$
3
  • $\begingroup$ Setting Attribute to Listable seems to be the problem. You can try with Function directly: g = Function[x, Piecewise[{{0.3*x, x < 0}, {x, x > 0}}], Listable] $\endgroup$
    – Anjan Kumar
    Commented May 22, 2017 at 4:53
  • $\begingroup$ @AnjanKumar Thank you,I edit my question. $\endgroup$
    – partida
    Commented May 22, 2017 at 5:12
  • $\begingroup$ Regarding the learned parameter of PReLU, I think you need to use ConstantArrayLayer for learned constants, and (I guess) ThreadingLayer to combine input from the constant and the "data" input layers $\endgroup$
    – Niki Estner
    Commented May 22, 2017 at 7:09

3 Answers 3

Reset to default
8
$\begingroup$

The documentation of ElementwiseLayer explicitly lists which functions are allowed, and UnitStep is not in this list, I believe that's why the function fails.

Simple workaround: Use a combination of functions from that list, like Ramp:

f = Ramp[#] - Ramp[-#]*0.3 &;

l = ElementwiseLayer[f]
l[{-1, -0.5, 0, 0.5, 1}]

{-0.3, -0.15, 0., 0.5, 1.}

Improve this answer
$\endgroup$
4
$\begingroup$

This is how to constract a PReLU

data = Thread[RandomReal[1, {100, 2}] -> RandomReal[1, {100, 3}]];
net = NetGraph[{5, ConstantArrayLayer["Output" -> 1], 
         ReplicateLayer[5], FlattenLayer[], 
         ThreadingLayer[Ramp[#1] - #2*Ramp[-#1] &], 3}, 
         {NetPort["Input"] -> 1 -> 5 -> 6, 2 -> 3 -> 4 -> 5}]

enter image description here

NetTrain[net, data, MaxTrainingRounds -> 100, BatchSize -> 32];
(*well done*)

In 11.2,mma provide more Advanced Activations Layers

enter image description here

Improve this answer
$\endgroup$
2
$\begingroup$

In 2020, the function ParametricRampLayer was introduced to implement leaky ReLU layers. The slope can be either hard-coded or learned.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.