Neural Network Simple task

Question

I'm trying to make a simple neural network to be able to find the y valeu given x1, x2 and x3. So first I made the training data:

trainingdata = 
 Dataset[{<|"x1" -> 0, "x2" -> 0, "x3" -> 1, "y" -> 0 |>, <|
    "x1" -> 0, "x2" -> 1, "x3" -> 1, "y" -> 1|>, <|"x1" -> 1, 
    "x2" -> 0, "x3" -> 1, "y" -> 1 |>, <|"x1" -> 1, "x2" -> 1, 
    "x3" -> 1, "y" -> 0 |>}]

Then I designed the net with 2 layers and 3 nodes in each one.

net = NetGraph[{CatenateLayer[], LinearLayer[1], LinearLayer[1], 
   LinearLayer[1], CatenateLayer[], CatenateLayer[], CatenateLayer[], 
   LinearLayer[1], LinearLayer[1], LinearLayer[1], CatenateLayer[], 
   LinearLayer[]}, {{NetPort["x1"], NetPort["x2"], NetPort["x3"]} -> 
    1 ->  {2, 3, 4} -> {5, 6, 7}, {5 -> 8}, {6 -> 9}, {7 -> 10}, {8, 
     9, 10} -> 11 -> 12 -> NetPort["y"]}, "x1" -> "Scalar", 
  "x2" -> "Scalar", "x3" -> "Scalar", "y" -> "Scalar"]

Then using NetTrain I trained the net:

trained = 
 NetTrain[net, trainingdata, MaxTrainingRounds -> 1500]

But even with more training rounds I still getting a pretty bad result, even with the cases that were in the training data, so I have the felling that I'm making something wrong here.

Obs: I know that using Classify I would get this correctly with easy, but I'm trying to learn how to use this part of neural nets in mathematica.

Answer 1

I suspect you are not defining the network that you need to solve the problem, as you have not introduced any nonlinear activation functions.

The type of network you have specified (concatenate 3 scalars, two width-3 layers, one width-1 layer) can be written in a simplified way using the LinearLayer arguments, instead of including many concatenations. Furthermore, one must specify the desired activation function (here I am using the Ramp function aka ReLU), which operates on each element

trainingdata = 
 Dataset[{<|"x1" -> 0, "x2" -> 0, "x3" -> 1, "y" -> 0|>, <|"x1" -> 0, 
    "x2" -> 1, "x3" -> 1, "y" -> 1|>, <|"x1" -> 1, "x2" -> 0, 
    "x3" -> 1, "y" -> 1|>, <|"x1" -> 1, "x2" -> 1, "x3" -> 1, 
    "y" -> 0|>}]

net = NetGraph[
   {CatenateLayer[], LinearLayer[3], Ramp, LinearLayer[3], Ramp, LinearLayer[1]}, 
   {{NetPort["x1"], NetPort["x2"], NetPort["x3"]} -> 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> NetPort["y"]}, 
  "x1" -> "Scalar", "x2" -> "Scalar", "x3" -> "Scalar", "y" -> "Scalar"]

trained = NetTrain[net, trainingdata]

(in this case, you could also use NetChain, which would further simplify the expression, but I have kept NetGraph to be consistent with your usage)

This correctly reproduces the training data:

trained[<|"x1" -> 0, "x2" -> 0, "x3" -> 1|>] (*0*)
trained[<|"x1" -> 0, "x2" -> 1, "x3" -> 1|>] (*1*)

In contrast, removing the activation functions causes this to fail)