11
$\begingroup$

I'm trying to reproduce the result giving by a Predict function with a Neural Network as the chosen method.

So, I'm training from this set:

trainingset = {1 -> 2, 2 -> 3, 3 -> 6, 4 -> 8};

and train a simple NN with just one HL with 1 linear neuron:

p = Predict[trainingset, Method -> {"NeuralNetwork", "HiddenLayers" -> {{1, "Linear"}}}]

By using Options I can have access to the resulting weights from:

Normal[Options[p][[1]]][[7]]

"Models" -> {<|"Method" -> "NeuralNetwork", "NeuralNetwork" -> 
<|"NeuronTypes" -> {"Linear", "Linear"}, "CostFunction" -> "SquaredCost", 
 "NumberOfNodes" -> {1, 1, 1}, "L1Regularization" -> 0, "L2Regularization" -> 0.1, 
 "DropOut" -> False, "Weights" -> {{{-0.9752752696739045}}, 
   {{-0.9752752696739043}}}, "Biases" -> {{-5.738324013344273*^-17}, 
   {-1.5516692498423558*^-17}}, "TrainingCostHistory" -> 
  {1.8283207510222368, 0.4788249850114622, 0.04910194009988826, 
   0.042738345120368906, 0.041599875787272395, 0.0410427527972165, 
   0.03861566085385098, 0.03761310711018569, 0.03599253089639499, 
   0.035775402536093924, 0.03573690274038513, 0.03573478075597261, 
   0.035734174870756356, 0.03573417449687247, 0.035734174496067674, 
   0.035734174496065876, 0.03573417449606588, 0.03573417449606589, 
   0.03573417449606589, 0.03573417449606589, 0.03573417449606589, 
   0.03573417449606589}, "TestCostHistory" -> {}|>, "EarlyStopping" -> False, 
"MaxIterations" -> 2250, "FeatureIndices" -> All, "DistributionData" -> 
{NormalDistribution, 0.1785506703441404}, "FeaturePreprocessor" -> 
MachineLearning`PackageScope`Preprocessor["Standardize", {{2.5}, {Sqrt[5/3]}}], 
"ExtractedFeatureNumber" -> 1|>}

They are both -0.9752. The predict function p gives me as a result:

 p[2]=3.73555

and now I'm trying to obtain this same result by calculation it by hand. So, since it is a linear NN, and because the biases are nearly zero, we should have.

output = (x * w1) * w2 = (2 * -0.9752) * -0.9752 = 1.90203

So, what am I doing wrong?

$\endgroup$
  • 1
    $\begingroup$ Are you sure you can just shove 2 in there directly? From your code it looks like features are standardized first. Practically all these options are undocumented too, I'd be interested in the solution as well. $\endgroup$ – Histograms May 28 '15 at 1:12
  • $\begingroup$ Not an answer, but another related question: Where can I find the documentation about predict using ANN? That is, what options are available? How do I specify the number and size of the hidden layers? How to specify the activation functions for the different layers?, etc. Thanks in advance. $\endgroup$ – Juan Flores Jan 11 '18 at 14:28
  • $\begingroup$ In version 12, this code does not seem to work anymore. Is there a way to obtain the weights from a NN in version 12? $\endgroup$ – Whelp Apr 29 at 8:43
4
$\begingroup$

As @Histograms mentioned in the comment, the features are standardized. Actually, both the input and output are standardized.

In the example, the mean and standard deviation of the input is:

input = {1, 2, 3, 4};
μ1 = Mean[input]
σ1 = StandardDeviation[input]
(* 5/2 *)
(* Sqrt[5/3] *)

so the linear function that standardized the input is

f1 = (# - μ1)/σ1 &

while the function that inverts this process is

g1 = μ1 + σ1 # &

The same for output:

output = {2, 3, 6, 8};
μ2 = Mean[output];
σ2 = StandardDeviation[output];  
f2 = (# - μ2)/σ2 &;
g2 = μ2 + σ2 # &;

So the prediction can be calculated as:

g2[(-0.975275)*(-0.975275)*f1[2]]
(* 3.73555 *)

The same as the value from Predict. More about processors can be found here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.