Neural network giving different answers

Question

I'm very new to machine learning and pretty new to Mathematica, trying to implement a simple neural network that acts as a "XOR" gate. Here's my code:

net = NetChain[{LinearLayer[2], LogisticSigmoid, LinearLayer[], 
LogisticSigmoid}];
trained = NetTrain[net, {{1, 1} -> 0, {1, .8} -> 0, {0, .2} -> 0, {0, 
1} -> 1, {1, 0} -> 1, {0, 0} -> 0}];
trained[{.2, .8}, None]

I'm confused because this code produces different outputs when I run it multiple times. I would have thought that Mathematica would just use some sort of least squares algorithm to compute weights based on my input, but it seems that Mathematica is doing something much more complicated. Can you explain why this is happening? And can you explain how I might eliminate this indeterminacy?

Answer 1

1. The weights and biases of neural networks must be initialized. Setting them all to zero is a bad idea from a machine learning point of view. So, often they are initialized randomly. See the Mathematica documentation for NetInitialize for more information or disabling random initialization.

2. Training networks often employ stochastic processes, such as ADAM Optimizer which is the default for Mathematica.

Therefore, initialization and optimization have randomness hidden in them, leading to different trained nets everytime you create and train them.