11
$\begingroup$

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha,but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

Application: this post used leayReLU[alpha_] := ElementwiseLayer[Ramp[#] - alpha*Ramp[-#] &]

$\endgroup$
  • $\begingroup$ Setting Attribute to Listable seems to be the problem. You can try with Function directly: g = Function[x, Piecewise[{{0.3*x, x < 0}, {x, x > 0}}], Listable] $\endgroup$ – Anjan Kumar May 22 '17 at 4:53
  • $\begingroup$ @AnjanKumar Thank you,I edit my question. $\endgroup$ – partida May 22 '17 at 5:12
  • $\begingroup$ Regarding the learned parameter of PReLU, I think you need to use ConstantArrayLayer for learned constants, and (I guess) ThreadingLayer to combine input from the constant and the "data" input layers $\endgroup$ – Niki Estner May 22 '17 at 7:09
7
$\begingroup$

The documentation of ElementwiseLayer explicitly lists which functions are allowed, and UnitStep is not in this list, I believe that's why the function fails.

Simple workaround: Use a combination of functions from that list, like Ramp:

f = Ramp[#] - Ramp[-#]*0.3 &;

l = ElementwiseLayer[f]
l[{-1, -0.5, 0, 0.5, 1}]

{-0.3, -0.15, 0., 0.5, 1.}

$\endgroup$
3
$\begingroup$

This is how to constract a PReLU

data = Thread[RandomReal[1, {100, 2}] -> RandomReal[1, {100, 3}]];
net = NetGraph[{5, ConstantArrayLayer["Output" -> 1], 
         ReplicateLayer[5], FlattenLayer[], 
         ThreadingLayer[Ramp[#1] - #2*Ramp[-#1] &], 3}, 
         {NetPort["Input"] -> 1 -> 5 -> 6, 2 -> 3 -> 4 -> 5}]

enter image description here

NetTrain[net, data, MaxTrainingRounds -> 100, BatchSize -> 32];
(*well done*)

In 11.2,mma provide more Advanced Activations Layers

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.