I want to have my own activation function, say a quadratic function, instead of Tanh, or Sigmoid etc. in Mathematica. How do I insert it in NetChain? e.g., how will the below code look for a new activation function?

net = NetChain[{10, Tanh, 10, Tanh, 10, Tanh, 10, Tanh, 1}, 
   "Input" -> "Scalar", "Output" -> "Scalar"]


  • 2
    $\begingroup$ Possibly duplicate question: How to make Advanced Activations Layers in Mathematica?. $\endgroup$
    – Silvia
    Commented May 1, 2019 at 4:45
  • $\begingroup$ I think you're looking for ElementwiseLayer. In v12, it also supports common activation functions like SELU etc. $\endgroup$ Commented May 1, 2019 at 5:13
  • $\begingroup$ It also supports those shorthand named functions in 11.3 (and 11.2, I think) :) $\endgroup$
    – Carl Lange
    Commented May 1, 2019 at 6:59

1 Answer 1


You want ElementwiseLayer. For example, here's how to define a leaky ReLU activation layer:

leakyRelu[α_] := ElementwiseLayer[
  If[# > 0, #, α #] &

We can see that this works:

Plot[leakyRelu[.3][x], {x, -1, 1}]

enter image description here

You might use it like so in your network:

net = NetChain[{10, leakyRelu[.3], 10, leakyRelu[.3], 10, leakyRelu[.3], 10, leakyRelu[.1], 1}, "Input" -> "Scalar", "Output" -> "Scalar"]]

You can add a reasonable amount of logic within the ElementwiseLayer, though there are a lot of symbols that won't work (for instance, you can't use ImageIdentify within your activation layer, but you can use Ramp, LogisticSigmoid, Tanh, ArcTan, ArcTanh, Sin, Sinh, ArcSin, ArcSinh, Cos, Cosh, ArcCos, ArcCosh, Log, Exp, Sqrt, Abs, Gamma, LogGamma... An incomplete list is in the documentation for ElementwiseLayer.

Regarding your comment with more information: Assuming you want to have a b and c configurable, you can do:

yourActivation[a_, b_, c_] := ElementwiseLayer[
  a #^2 + b # + c &

and then you can use it in a NetChain like so:

net = NetChain[{10, yourActivation[.3, .4, .1], 10, 1}, 
  "Input" -> "Scalar", "Output" -> "Scalar"]
  • $\begingroup$ what is alpha here? What is the expression of the function you are using? I am looking for activation function of f[x] = a x^2 + b x + c. $\endgroup$
    – jjal
    Commented May 2, 2019 at 0:29
  • $\begingroup$ Alpha is just a configurable weight - look up the Leaky ReLU activation function for more information about it. What is a, b, and c in your function? $\endgroup$
    – Carl Lange
    Commented May 2, 2019 at 7:19
  • $\begingroup$ @jjal See my edit $\endgroup$
    – Carl Lange
    Commented May 2, 2019 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.