Skip to main content
Mathematica

Return to Question

added 224 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha，but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

Application: this post used leayReLU[alpha_] := ElementwiseLayer[Ramp[#] - alpha*Ramp[-#] &]

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha，but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha，but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

Application: this post used leayReLU[alpha_] := ElementwiseLayer[Ramp[#] - alpha*Ramp[-#] &]

Tweeted twitter.com/StackMma/status/866599020613029889
deleted 211 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48

I want to try  LeakyReLU LayerAdvanced Activations Layers in Mathematica,but not found.

In keras,the LeakyReLU be defined like this:

enter image description here

So I try to use ElementwiseLayer to implement it.

g = Function[x, Piecewise[{{0.3(*default value*)*x, x < 0}, {x, x >= 0}}], Listable];
Plot[g[x], {x, -2, 2}]
g[{-1, -0.5, 0, 0.5, 1}]

enter image description here

But ElementwiseLayer[g] failed.

Then I try:

f = UnitStep[#]*# + UnitStep[-#]*0.3# &;
Plot[f[x], {x, -2, 2}]
f[{-1, -0.5, 0, 0.5, 1}]

enter image description here

It also failed.

enter image description here

It's confused meThank you @nikie,so how to make LeakyReLU or other advanced activations layers in Mathematica?

What about PReLULeakyReLU,ELU,ThresholdedReLU?

Ps: Thank you @nikie can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

But PReLU has a learned parameter alpha so，but I have no ideadon't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

I want to try  LeakyReLU Layer in Mathematica,but not found.

In keras,the LeakyReLU be defined like this:

enter image description here

So I try to use ElementwiseLayer to implement it.

g = Function[x, Piecewise[{{0.3(*default value*)*x, x < 0}, {x, x >= 0}}], Listable];
Plot[g[x], {x, -2, 2}]
g[{-1, -0.5, 0, 0.5, 1}]

enter image description here

But ElementwiseLayer[g] failed.

Then I try:

f = UnitStep[#]*# + UnitStep[-#]*0.3# &;
Plot[f[x], {x, -2, 2}]
f[{-1, -0.5, 0, 0.5, 1}]

enter image description here

It also failed.

enter image description here

It's confused me,so how to make LeakyReLU or other advanced activations layers in Mathematica?

What about PReLU,ELU,ThresholdedReLU?

Ps: Thank you @nikie

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

But PReLU has a learned parameter alpha so I have no idea...

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha，but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

added 73 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48

I want to try LeakyReLU Layer in Mathematica,but not found.

In keras,the LeakyReLU be defined like this:

enter image description here

So I try to use ElementwiseLayer to implement it.

g = Function[x, Piecewise[{{0.3(*default value*)*x, x < 0}, {x, x >= 0}}], Listable];
Plot[g[x], {x, -2, 2}]
g[{-1, -0.5, 0, 0.5, 1}]

enter image description here

But ElementwiseLayer[g] failed.

Then I try:

f = UnitStep[#]*# + UnitStep[-#]*0.3# &;
Plot[f[x], {x, -2, 2}]
f[{-1, -0.5, 0, 0.5, 1}]

enter image description here

It also failed.

enter image description here

It's confused me,so how to make LeakyReLU or other advanced activations layers in Mathematica?

What about PReLU,ELU,ThresholdedReLU?

Ps: Thank you @nikie

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

But PReLU has a learned parameter alpha so I have no idea...

I want to try LeakyReLU Layer in Mathematica,but not found.

In keras,the LeakyReLU be defined like this:

enter image description here

So I try to use ElementwiseLayer to implement it.

g = Function[x, Piecewise[{{0.3(*default value*)*x, x < 0}, {x, x >= 0}}], Listable];
Plot[g[x], {x, -2, 2}]
g[{-1, -0.5, 0, 0.5, 1}]

enter image description here

But ElementwiseLayer[g] failed.

Then I try:

f = UnitStep[#]*# + UnitStep[-#]*0.3# &;
Plot[f[x], {x, -2, 2}]
f[{-1, -0.5, 0, 0.5, 1}]

enter image description here

It also failed.

enter image description here

It's confused me,so how to make LeakyReLU or other advanced activations layers in Mathematica?

What about PReLU,ELU,ThresholdedReLU?

Ps: Thank you @nikie

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

I want to try LeakyReLU Layer in Mathematica,but not found.

In keras,the LeakyReLU be defined like this:

enter image description here

So I try to use ElementwiseLayer to implement it.

g = Function[x, Piecewise[{{0.3(*default value*)*x, x < 0}, {x, x >= 0}}], Listable];
Plot[g[x], {x, -2, 2}]
g[{-1, -0.5, 0, 0.5, 1}]

enter image description here

But ElementwiseLayer[g] failed.

Then I try:

f = UnitStep[#]*# + UnitStep[-#]*0.3# &;
Plot[f[x], {x, -2, 2}]
f[{-1, -0.5, 0, 0.5, 1}]

enter image description here

It also failed.

enter image description here

It's confused me,so how to make LeakyReLU or other advanced activations layers in Mathematica?

What about PReLU,ELU,ThresholdedReLU?

Ps: Thank you @nikie

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

But PReLU has a learned parameter alpha so I have no idea...

added 45 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48
added 45 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48
deleted 13 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48
deleted 55 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48
deleted 3 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48
added 17 characters in body
Source Link
partida
partida
  • 6.9k
  • 24
  • 48
Source Link
partida
partida
  • 6.9k
  • 24
  • 48