I am trying to build a simple neural network, with an input layer of 250 nodes, 2 'hidden' layers each of 100 nodes, and an output layer with 1 node. I want each node to take in the linear combination of every node in the previous layer and perform the ReLU function. So far, my code is

layer1 = LinearLayer[100,"Input"->250]
layer2 = LinearLayer[100,"Input"->100]
layer3 = LinearLayer[1,"Input"->100]


, but this is giving me an extra output layer and I am not sure how to encorporate the ReLU function.

  • $\begingroup$ Just riffle them: NetChain[{layer1,ElementwiseLayer["ReLU"],layer2,ElementwiseLayer["ReLU"],layer3}] $\endgroup$ – M.R. Aug 28 '18 at 16:21
  • $\begingroup$ Ok. I am confused though, because it looks like ElementwiseLayer["ReLU"] is adding another layer, whereas I want the ReLU function to be applied to each node in the layer $\endgroup$ – pmac Aug 28 '18 at 16:28
  • 1
    $\begingroup$ And when you read about ElementwiseLayer, what does the documentation say that it does? It is the very thing you are looking for. I think it could help to think about neural networks, at least in Wolfram Language, not as layers with nodes but to think of each layer as a function. You're passing a tensor through a series of functions. It is completely natural from this point of view that activation functions are layers. $\endgroup$ – C. E. Aug 28 '18 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.