4
$\begingroup$

I am trying to convert neural networks from a book to Wolfram language. The book has this example with the quote

two layers of 10 neurons and a dropout of 0.2 for each

 def create_mlp_2layer(): 
 model_arch = [ 
 Dense(10, activation="relu"), 
 Dropout(0.2), 
 Dense(10, activation="relu"), 
 Dropout(0.2), 
 Dense(1) 
 ] 
 model = Sequential(model_arch) 
 model.compile(optimizer="adam", loss ='mse') 
 return model 

I have used Predict, Classify, and I am conversant with data pre-processing to feed data appropriately. However, I would like to learn this "translation".

As far as I was able to figure out, relu would likely become Ramp[], Dropout(0.2) would become DropoutLayer[0.2],however, I am not clear about how to put everything together. The net should return a real number as the output.

The input is a vector of length 167 which is mostly 0's and a few 1's.

I tried to hack this together, however, I am lost.

NetChain[{LinearLayer[167], ElementwiseLayer[Ramp], DropoutLayer[0.2], 
  ElementwiseLayer[Ramp], DropoutLayer[0.2], LinearLayer[1]}]

Using NetInitialize on this complains that

Cannot initialize net: unspecified or partially specified shape for array "Weights" of first layer.

It would be great if anyone can show how to translate this to WL syntax to prepare the NN.

Thank you,

$\endgroup$

1 Answer 1

5
$\begingroup$

When it cannot be inferred from previous layers in a net, the option "Input"->shape can be used to fix the input of LinearLayer. You need to specify the shape of input and output like this:

net = NetChain[{LinearLayer[10, "Input" -> 167], 
ElementwiseLayer["ReLU"], DropoutLayer[0.2], LinearLayer[10], 
ElementwiseLayer["ReLU"], DropoutLayer[0.2], 
LinearLayer["Output" -> "Real"]}];
net = NetInitialize[net]
(*Input: vector:size 167, Output: real number*)

Possible forms for shape can be found here: LinearLayer

Then you can train your network:

lstP = Thread[Table[RandomReal[{0, 10}, 167], 100] -> 1];
lstN = Thread[Table[RandomReal[{-10, 0}, 167], 100] -> -1];
trainingSet = Join[lstP, lstN];
trainedNet = 
NetTrain[net, trainingSet, Method -> "ADAM", 
LossFunction -> MeanSquaredLossLayer[], MaxTrainingRounds -> 2000]

Evaluate some samples:

testSample1 = Table[RandomReal[{0, 10}, 167], 5];
testSample2 = Table[RandomReal[{-10, 0}, 167], 5];


trainedNet /@ testSample1
(*{0.798105, 0.680741, 0.715918, 0.639813, 0.724121}*)

trainedNet /@ testSample2
(*{-0.908868, -0.697567, -0.809703, -0.907407, -0.863312}*)

Notice that if you use MeanSquaredLossLayer, you may need to create an additional output port to use NetMeasurements which can be found here: NetMeasurements-Examples-Possible_Issues

$\endgroup$
4
  • $\begingroup$ Thanks for your detailed answer. I have a follow-up question. Do we not need another LinearLayer[10] before the second DropoutLayer[0.2]? I am trying to build correspondence and want to make sure I understand your steps. Thanks again! $\endgroup$
    – bhopshang
    Jan 29 at 17:12
  • $\begingroup$ The function of Dropout layer is to randomly drop units (along with their connections) from the neural network during training, so you may use it after a layer. In Mathematica, DropoutLayer normally infers the dimensions of its input from its context which in this example, the previous layer. So another LinearLayer is recommended since dropout the same data twice is redundant. $\endgroup$ Jan 29 at 19:05
  • $\begingroup$ So, this should be the net? Could you please edit your answer if that is the case? I will accept the answer once that's done. Thank you so much! NetChain[{ LinearLayer[167, "Input" -> 167], ElementwiseLayer["ReLU"], DropoutLayer[0.2], LinearLayer[10], ElementwiseLayer["ReLU"], LinearLayer[10], DropoutLayer[0.2], LinearLayer["Output" -> "Real"] }] $\endgroup$
    – bhopshang
    Jan 29 at 21:25
  • $\begingroup$ I edited the answer since there are only two hidden layers to your question, the first layer should be modified as LinearLayer[10, "Input" -> 167]. The MaxTraining Rounds increases as the total number of neural units decrease. $\endgroup$ Jan 30 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.