I need a specific custom activation function; how can I implement it with Piecewise?

This is what I've tried (using the recommendation from this question):

p = Function[x, Piecewise[{{Exp[x] - (2 + E)/(2 E), x < -1}, {x/2, x < 0}, {x, 
 x > 0}}], Listable]

enter image description here

Is there a systematic workaround to handle the conversion from an aribitrary Piecewise function into an ElementwiseLayer?


Let me give a general method for re-expressing a Piecewise[] expression. This hinges on two things. First:

FullSimplify[1 - Sign[Ramp[-x]] == UnitStep[x], x ∈ Reals]

The second part is that in principle, any piecewise expression can be re-expressed in terms of UnitStep[]; in particular, there is the undocumented function Simplify`PWToUnitStep[] (see here) for performing the conversion.

With these two considerations (and using a less trivial example):

Simplify`PWToUnitStep[Piecewise[{{2 - #, # <= 0}, {# + 1, # > 0}}] &[x]] /.
UnitStep -> (1 - Sign[Ramp[-#]] &)
   (2 - x) (1 - Sign[Ramp[x]]) + (1 + x) Sign[Ramp[x]]

and now one can do ElementwiseLayer[Function[x, (2 - x) (1 - Sign[Ramp[x]]) + (1 + x) Sign[Ramp[x]]]].

| improve this answer | |
  • $\begingroup$ thanks for a simple generalizable explanation! $\endgroup$ – user5601 Mar 20 '18 at 14:55

From the documentation of ElementwiseLayer:

The function f can be any one of the following: Ramp, LogisticSigmoid, Tanh, ArcTan, ArcTanh, Sin, Sinh, ArcSin, ArcSinh, Cos, Cosh, ArcCos, ArcCosh, Log, Exp, Sqrt, Abs, Gamma, LogGamma.

In general, f can be any object that when applied to a single argument gives any combination of Ramp, LogisticSigmoid, etc., together with Plus, Subtract, Times, Divide, Power, Min, Max, Clip, and numbers.

So you'll need to use something like ElementwiseLayer[Ramp[#] - Ramp[-#] &]

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.