2
$\begingroup$

I'm very new to machine learning and pretty new to Mathematica, trying to implement a simple neural network that acts as a "XOR" gate. Here's my code:

net = NetChain[{LinearLayer[2], LogisticSigmoid, LinearLayer[], 
LogisticSigmoid}];
trained = NetTrain[net, {{1, 1} -> 0, {1, .8} -> 0, {0, .2} -> 0, {0, 
1} -> 1, {1, 0} -> 1, {0, 0} -> 0}];
trained[{.2, .8}, None]

I'm confused because this code produces different outputs when I run it multiple times. I would have thought that Mathematica would just use some sort of least squares algorithm to compute weights based on my input, but it seems that Mathematica is doing something much more complicated. Can you explain why this is happening? And can you explain how I might eliminate this indeterminacy?

$\endgroup$
  • 1
    $\begingroup$ I'm not sure how to answer this question without basically explaining how neural network training works - and there already are many good tutorials on that. In a nutshell: It's using (a form of) gradient descent from a random starting point. If you don't want that, neural networks are probably not the technology you want. $\endgroup$ – Niki Estner Nov 28 '17 at 16:55
  • 2
    $\begingroup$ There is some stochasticity to it, yes. $\endgroup$ – Daniel Lichtblau Nov 28 '17 at 21:22
  • $\begingroup$ Placing BlockRandom around the call to NetTrain makes things deterministic. $\endgroup$ – Chip Hurst Nov 29 '17 at 3:16
  • $\begingroup$ @ChipHurst: in general, no. First, for GPU training, BlockRandom has no effect on cuDNN primitives at the moment. Second, there will almost always be some stochasticity due to all of our numerics being completely parallelized (things like numeric addition are not associative, meaning slightly different answers are gotten if parallel threads do things in different orders). $\endgroup$ – Sebastian Dec 7 '17 at 17:41
  • $\begingroup$ @Sebastian I guess I left this comment because Equal @@ Table[BlockRandom[(* OP's code *)], 10] returns True on my machine. $\endgroup$ – Chip Hurst Dec 7 '17 at 22:22
3
$\begingroup$
  1. The weights and biases of neural networks must be initialized. Setting them all to zero is a bad idea from a machine learning point of view. So, often they are initialized randomly. See the Mathematica documentation for NetInitialize for more information or disabling random initialization.

  2. Training networks often employ stochastic processes, such as ADAM Optimizer which is the default for Mathematica.

Therefore, initialization and optimization have randomness hidden in them, leading to different trained nets everytime you create and train them.

$\endgroup$
  • $\begingroup$ If you need a deterministic example of a neural network, I can cook up something for you. Just let me know. $\endgroup$ – Miladiouss Nov 29 '17 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.