13
$\begingroup$

How can I manipulate the architecture and problems of a NeuralNetwork in Predict or Classify? For example, running the following code shows a number of properties of the generated network.

data = {Range[1, 100], RandomReal[NormalDistribution[0, 0.01], {100}] - Range[0, 0.1, 0.1/(100 - 1)]} // Transpose;
data = Rule @@@ data;
model = Predict[data, Method -> "NeuralNetwork"];
PredictorInformation[model]

How can I manipulate the options such as the regularization coefficients, the number of hidden layers, the number of hidden nodes, and the hidden layer activation functions?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
15
$\begingroup$

As already mentioned, options are accessed via:

Method -> {"NeuralNetwork", "L2Regularization" -> 0.12,  "HiddenLayers" -> {4, 3, 3}}

To manually set the activation functions of the first and second hidden layers:

"HiddenLayers"-> {{4, "RectifiedLinear"}, {3, "Tanh"}, 3}

Currently, the following activation functions are supported:

{"LogisticSigmoid", "RectifiedLinear", "Tanh", "SoftRectifiedLinear", "Linear"}

Non-zero "L1Regularization" is not yet supported.

The usual proviso: These options are undocumented functionality whose form may change substantially in future versions.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Great, thanks. Another clarification: is it possible to specify the activation of the output layer? $\endgroup$
    – mdeceglie
    Commented Nov 4, 2014 at 23:27
  • $\begingroup$ Not directly. For the output layer, Classify uses softmax (to get class probabilities) and Predict uses linear. But Classify and Predict give you the outputs, so you can take this and apply a custom activation function. Or do you want to change the loss function used for training? $\endgroup$
    – Sebastian
    Commented Nov 5, 2014 at 0:56
  • $\begingroup$ So, it is indeed, as I assumed, a specific "HiddenLayers" format that is needed, but I only tried to specify functions instead of names as strings :-( $\endgroup$
    – Sjoerd C. de Vries
    Commented Nov 5, 2014 at 11:26
  • $\begingroup$ Setting "L2Regularization" seems to hang the kernel in 10.4, has the option been changed? $\endgroup$
    – Silvia
    Commented Mar 17, 2016 at 18:16
14
$\begingroup$

At least 

Method -> {"NeuralNetwork", "L2Regularization" -> 0.01,  "HiddenLayers" -> {4, 3, 3}}

seems to work. "L1Regularization" doesn't seem to be settable. Haven't found out about activation functions yet.

Mathematica graphics

Update

More options can be found using

Options@MachineLearning`PackageScope`NeuralNetworkPredictor

{"BinaryEncoder" -> "Identity", "ClassNumber" -> Automatic, "CrossValidation" -> Automatic, "DataSize" -> All, "EarlyStopping" -> Automatic, "HiddenLayers" -> Automatic, "L2Regularization" -> Automatic, "LearningTask" -> "Predict", MaxIterations -> Automatic, NominalVariables -> Automatic, PerformanceGoal -> Automatic, "PrintCost" -> False, "OptimizationMethod" -> Automatic, "WeightInitializationMethod" -> Automatic}

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks, this is a great start. I would really like to be able to set the activation functions though, so if you can figure that out, this will be the accepted answer. $\endgroup$
    – mdeceglie
    Commented Nov 4, 2014 at 16:02
  • $\begingroup$ @Italianice There really doesn't seem to be a named option to specify activation functions, but I have reasons to believe that it should be possible to do that using a particular format for the "HiddenLayers" option specification. I couldn't find out what would do the trick though. $\endgroup$
    – Sjoerd C. de Vries
    Commented Nov 4, 2014 at 21:24
  • $\begingroup$ Does running Options[model] provide any clues? The output contains "NeuronTypes" -> {"Tanh", "Tanh", "Linear"}, but it is inside some nested association structure that I don't yet understand. $\endgroup$
    – mdeceglie
    Commented Nov 4, 2014 at 22:41
  • $\begingroup$ Do you know how can I change "CrossValidation"? $\endgroup$
    – Miguel
    Commented May 26, 2015 at 11:50
  • $\begingroup$ @Miguel You might want to look into MachineLearning`PackageScope`ParseCrossValidationOptions . It seems to list its options, but I can't get it to work. $\endgroup$
    – Sjoerd C. de Vries
    Commented May 26, 2015 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.