I want to analize a network such that all neurons of its hidden layers output LogisticSigmoid[w.x+b], which is one of the most common ouputs analized in a neuron. Is there a way to combine LinearLayer[] that gives output w.x+b and the LogisticSigmoid function? I looked for an option of changing the activation function in LinearLayer[] but I did not find it.

Note: I do not want to use twice ammount of hidden layers by first applying LinearLayer[] and then ElementWiseLayer[LogisticSigmoid], I would like the network to be more compact. How would I do that?

  • 1
    $\begingroup$ But ElementWiseLayer doesn't contain any parameters, it's the same as changing activation function for LinearLayer. You don't even have to apply ElementWiseLayer, just LogisticSigmoid works. $\endgroup$ – swish Apr 10 '18 at 17:45

I guess maybe you know the function in Keras:

model = Sequential()
model.add(Dense(5, activation='sigmoid', input_shape=(10,)))
model.add(Dense(5, activation='sigmoid'))
model.add(Dense(5, activation='sigmoid'))

Note, that is Dense layer rather than linear layer, so it have the option that can define the activation function.

In MMA, that is simple.

Dense[outDims_, "activation" -> act_] := 
  Switch[act, "tanh", ElementwiseLayer[Tanh], "relu", 
   ElementwiseLayer[Ramp], "sigmoid", 

NetChain[{Dense[5, "activation" -> "sigmoid"], 
          Dense[5, "activation" -> "sigmoid"], 
          Dense[5, "activation" -> "sigmoid"]}, "Input" -> 10]

enter image description here

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.