# How do I create my own activation function in NetChain in Mathematica?

I want to have my own activation function, say a quadratic function, instead of Tanh, or Sigmoid etc. in Mathematica. How do I insert it in NetChain? e.g., how will the below code look for a new activation function?

net = NetChain[{10, Tanh, 10, Tanh, 10, Tanh, 10, Tanh, 1},
"Input" -> "Scalar", "Output" -> "Scalar"]


Thanks.

• Possibly duplicate question: How to make Advanced Activations Layers in Mathematica?. Commented May 1, 2019 at 4:45
• I think you're looking for ElementwiseLayer. In v12, it also supports common activation functions like SELU etc. Commented May 1, 2019 at 5:13
• It also supports those shorthand named functions in 11.3 (and 11.2, I think) :) Commented May 1, 2019 at 6:59

You want ElementwiseLayer. For example, here's how to define a leaky ReLU activation layer:

leakyRelu[α_] := ElementwiseLayer[
If[# > 0, #, α #] &
]


We can see that this works:

Plot[leakyRelu[.3][x], {x, -1, 1}]


You might use it like so in your network:

net = NetChain[{10, leakyRelu[.3], 10, leakyRelu[.3], 10, leakyRelu[.3], 10, leakyRelu[.1], 1}, "Input" -> "Scalar", "Output" -> "Scalar"]]


You can add a reasonable amount of logic within the ElementwiseLayer, though there are a lot of symbols that won't work (for instance, you can't use ImageIdentify within your activation layer, but you can use Ramp, LogisticSigmoid, Tanh, ArcTan, ArcTanh, Sin, Sinh, ArcSin, ArcSinh, Cos, Cosh, ArcCos, ArcCosh, Log, Exp, Sqrt, Abs, Gamma, LogGamma... An incomplete list is in the documentation for ElementwiseLayer.

Regarding your comment with more information: Assuming you want to have a b and c configurable, you can do:

yourActivation[a_, b_, c_] := ElementwiseLayer[
a #^2 + b # + c &
]


and then you can use it in a NetChain like so:

net = NetChain[{10, yourActivation[.3, .4, .1], 10, 1},
"Input" -> "Scalar", "Output" -> "Scalar"]

• what is alpha here? What is the expression of the function you are using? I am looking for activation function of f[x] = a x^2 + b x + c.
– jjal
Commented May 2, 2019 at 0:29
• Alpha is just a configurable weight - look up the Leaky ReLU activation function for more information about it. What is a, b, and c in your function? Commented May 2, 2019 at 7:19
• @jjal See my edit Commented May 2, 2019 at 7:34