Constructing neural networks from Keras example

Question

I am trying to convert neural networks from a book to Wolfram language. The book has this example with the quote

two layers of 10 neurons and a dropout of 0.2 for each

 def create_mlp_2layer(): 
 model_arch = [ 
 Dense(10, activation="relu"), 
 Dropout(0.2), 
 Dense(10, activation="relu"), 
 Dropout(0.2), 
 Dense(1) 
 ] 
 model = Sequential(model_arch) 
 model.compile(optimizer="adam", loss ='mse') 
 return model 

I have used Predict, Classify, and I am conversant with data pre-processing to feed data appropriately. However, I would like to learn this "translation".

As far as I was able to figure out, relu would likely become Ramp[], Dropout(0.2) would become DropoutLayer[0.2],however, I am not clear about how to put everything together. The net should return a real number as the output.

The input is a vector of length 167 which is mostly 0's and a few 1's.

I tried to hack this together, however, I am lost.

NetChain[{LinearLayer[167], ElementwiseLayer[Ramp], DropoutLayer[0.2], 
  ElementwiseLayer[Ramp], DropoutLayer[0.2], LinearLayer[1]}]

Using NetInitialize on this complains that

Cannot initialize net: unspecified or partially specified shape for array "Weights" of first layer.

It would be great if anyone can show how to translate this to WL syntax to prepare the NN.

Thank you,

Answer 1

When it cannot be inferred from previous layers in a net, the option "Input"->shape can be used to fix the input of LinearLayer. You need to specify the shape of input and output like this:

net = NetChain[{LinearLayer[10, "Input" -> 167], 
ElementwiseLayer["ReLU"], DropoutLayer[0.2], LinearLayer[10], 
ElementwiseLayer["ReLU"], DropoutLayer[0.2], 
LinearLayer["Output" -> "Real"]}];
net = NetInitialize[net]
(*Input: vector:size 167, Output: real number*)

Possible forms for shape can be found here: LinearLayer

Then you can train your network:

lstP = Thread[Table[RandomReal[{0, 10}, 167], 100] -> 1];
lstN = Thread[Table[RandomReal[{-10, 0}, 167], 100] -> -1];
trainingSet = Join[lstP, lstN];
trainedNet = 
NetTrain[net, trainingSet, Method -> "ADAM", 
LossFunction -> MeanSquaredLossLayer[], MaxTrainingRounds -> 2000]

Evaluate some samples:

testSample1 = Table[RandomReal[{0, 10}, 167], 5];
testSample2 = Table[RandomReal[{-10, 0}, 167], 5];


trainedNet /@ testSample1
(*{0.798105, 0.680741, 0.715918, 0.639813, 0.724121}*)

trainedNet /@ testSample2
(*{-0.908868, -0.697567, -0.809703, -0.907407, -0.863312}*)

Notice that if you use MeanSquaredLossLayer, you may need to create an additional output port to use NetMeasurements which can be found here: NetMeasurements-Examples-Possible_Issues