ElementwiseLayer with Piecewise?

Question

I need a specific custom activation function; how can I implement it with Piecewise?

This is what I've tried (using the recommendation from this question):

p = Function[x, Piecewise[{{Exp[x] - (2 + E)/(2 E), x < -1}, {x/2, x < 0}, {x, 
 x > 0}}], Listable]
ElementwiseLayer[p]

Is there a systematic workaround to handle the conversion from an aribitrary Piecewise function into an ElementwiseLayer?

Answer 1

Let me give a general method for re-expressing a Piecewise[] expression. This hinges on two things. First:

FullSimplify[1 - Sign[Ramp[-x]] == UnitStep[x], x ∈ Reals]
   True

The second part is that in principle, any piecewise expression can be re-expressed in terms of UnitStep[]; in particular, there is the undocumented function Simplify`PWToUnitStep[] (see here) for performing the conversion.

With these two considerations (and using a less trivial example):

Simplify`PWToUnitStep[Piecewise[{{2 - #, # <= 0}, {# + 1, # > 0}}] &[x]] /.
UnitStep -> (1 - Sign[Ramp[-#]] &)
   (2 - x) (1 - Sign[Ramp[x]]) + (1 + x) Sign[Ramp[x]]

and now one can do ElementwiseLayer[Function[x, (2 - x) (1 - Sign[Ramp[x]]) + (1 + x) Sign[Ramp[x]]]].

Answer 2

From the documentation of ElementwiseLayer:

The function f can be any one of the following: Ramp, LogisticSigmoid, Tanh, ArcTan, ArcTanh, Sin, Sinh, ArcSin, ArcSinh, Cos, Cosh, ArcCos, ArcCosh, Log, Exp, Sqrt, Abs, Gamma, LogGamma.

In general, f can be any object that when applied to a single argument gives any combination of Ramp, LogisticSigmoid, etc., together with Plus, Subtract, Times, Divide, Power, Min, Max, Clip, and numbers.

So you'll need to use something like ElementwiseLayer[Ramp[#] - Ramp[-#] &]