3
$\begingroup$

I'm very new to machine learning and pretty new to Mathematica, trying to implement a simple neural network that acts as a "XOR" gate. Here's my code:

net = NetChain[{LinearLayer[2], LogisticSigmoid, LinearLayer[], 
LogisticSigmoid}];
trained = NetTrain[net, {{1, 1} -> 0, {1, .8} -> 0, {0, .2} -> 0, {0, 
1} -> 1, {1, 0} -> 1, {0, 0} -> 0}];
trained[{.2, .8}, None]

I'm confused because this code produces different outputs when I run it multiple times. I would have thought that Mathematica would just use some sort of least squares algorithm to compute weights based on my input, but it seems that Mathematica is doing something much more complicated. Can you explain why this is happening? And can you explain how I might eliminate this indeterminacy?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ I'm not sure how to answer this question without basically explaining how neural network training works - and there already are many good tutorials on that. In a nutshell: It's using (a form of) gradient descent from a random starting point. If you don't want that, neural networks are probably not the technology you want. $\endgroup$
    – Niki Estner
    Nov 28, 2017 at 16:55
  • 2
    $\begingroup$ There is some stochasticity to it, yes. $\endgroup$
    – Daniel Lichtblau
    Nov 28, 2017 at 21:22
  • $\begingroup$ Placing BlockRandom around the call to NetTrain makes things deterministic. $\endgroup$
    – Greg Hurst
    Nov 29, 2017 at 3:16
  • $\begingroup$ @ChipHurst: in general, no. First, for GPU training, BlockRandom has no effect on cuDNN primitives at the moment. Second, there will almost always be some stochasticity due to all of our numerics being completely parallelized (things like numeric addition are not associative, meaning slightly different answers are gotten if parallel threads do things in different orders). $\endgroup$
    – Sebastian
    Dec 7, 2017 at 17:41
  • $\begingroup$ @Sebastian I guess I left this comment because Equal @@ Table[BlockRandom[(* OP's code *)], 10] returns True on my machine. $\endgroup$
    – Greg Hurst
    Dec 7, 2017 at 22:22

1 Answer 1

Reset to default
4
$\begingroup$

  1. The weights and biases of neural networks must be initialized. Setting them all to zero is a bad idea from a machine learning point of view. So, often they are initialized randomly. See the Mathematica documentation for NetInitialize for more information or disabling random initialization.

  2. Training networks often employ stochastic processes, such as ADAM Optimizer which is the default for Mathematica.

Therefore, initialization and optimization have randomness hidden in them, leading to different trained nets everytime you create and train them.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If you need a deterministic example of a neural network, I can cook up something for you. Just let me know. $\endgroup$
    – Miladiouss
    Nov 29, 2017 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.