3
$\begingroup$

I want to have my own activation function, say a quadratic function, instead of Tanh, or Sigmoid etc. in Mathematica. How do I insert it in NetChain? e.g., how will the below code look for a new activation function?

net = NetChain[{10, Tanh, 10, Tanh, 10, Tanh, 10, Tanh, 1}, 
   "Input" -> "Scalar", "Output" -> "Scalar"]

Thanks.

$\endgroup$
  • 2
    $\begingroup$ Possibly duplicate question: How to make Advanced Activations Layers in Mathematica?. $\endgroup$ – Silvia May 1 at 4:45
  • $\begingroup$ I think you're looking for ElementwiseLayer. In v12, it also supports common activation functions like SELU etc. $\endgroup$ – Niki Estner May 1 at 5:13
  • $\begingroup$ It also supports those shorthand named functions in 11.3 (and 11.2, I think) :) $\endgroup$ – Carl Lange May 1 at 6:59
1
$\begingroup$

You want ElementwiseLayer. For example, here's how to define a leaky ReLU activation layer:

leakyRelu[α_] := ElementwiseLayer[
  If[# > 0, #, α #] &
  ]

We can see that this works:

Plot[leakyRelu[.3][x], {x, -1, 1}]

enter image description here

You might use it like so in your network:

net = NetChain[{10, leakyRelu[.3], 10, leakyRelu[.3], 10, leakyRelu[.3], 10, leakyRelu[.1], 1}, "Input" -> "Scalar", "Output" -> "Scalar"]]

You can add a reasonable amount of logic within the ElementwiseLayer, though there are a lot of symbols that won't work (for instance, you can't use ImageIdentify within your activation layer, but you can use Ramp, LogisticSigmoid, Tanh, ArcTan, ArcTanh, Sin, Sinh, ArcSin, ArcSinh, Cos, Cosh, ArcCos, ArcCosh, Log, Exp, Sqrt, Abs, Gamma, LogGamma... An incomplete list is in the documentation for ElementwiseLayer.


Regarding your comment with more information: Assuming you want to have a b and c configurable, you can do:

yourActivation[a_, b_, c_] := ElementwiseLayer[
  a #^2 + b # + c &
  ]

and then you can use it in a NetChain like so:

net = NetChain[{10, yourActivation[.3, .4, .1], 10, 1}, 
  "Input" -> "Scalar", "Output" -> "Scalar"]
$\endgroup$
  • $\begingroup$ what is alpha here? What is the expression of the function you are using? I am looking for activation function of f[x] = a x^2 + b x + c. $\endgroup$ – jjal May 2 at 0:29
  • $\begingroup$ Alpha is just a configurable weight - look up the Leaky ReLU activation function for more information about it. What is a, b, and c in your function? $\endgroup$ – Carl Lange May 2 at 7:19
  • $\begingroup$ @jjal See my edit $\endgroup$ – Carl Lange May 2 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.