# How to change NeuralNetwork options?

How can I manipulate the architecture and problems of a NeuralNetwork in Predict or Classify? For example, running the following code shows a number of properties of the generated network.

data = {Range[1, 100], RandomReal[NormalDistribution[0, 0.01], {100}] - Range[0, 0.1, 0.1/(100 - 1)]} // Transpose;
data = Rule @@@ data;
model = Predict[data, Method -> "NeuralNetwork"];
PredictorInformation[model]


How can I manipulate the options such as the regularization coefficients, the number of hidden layers, the number of hidden nodes, and the hidden layer activation functions?

As already mentioned, options are accessed via:

Method -> {"NeuralNetwork", "L2Regularization" -> 0.12,  "HiddenLayers" -> {4, 3, 3}}


To manually set the activation functions of the first and second hidden layers:

"HiddenLayers"-> {{4, "RectifiedLinear"}, {3, "Tanh"}, 3}


Currently, the following activation functions are supported:

{"LogisticSigmoid", "RectifiedLinear", "Tanh", "SoftRectifiedLinear", "Linear"}


Non-zero "L1Regularization" is not yet supported.

The usual proviso: These options are undocumented functionality whose form may change substantially in future versions.

• Great, thanks. Another clarification: is it possible to specify the activation of the output layer? – mdeceglie Nov 4 '14 at 23:27
• Not directly. For the output layer, Classify uses softmax (to get class probabilities) and Predict uses linear. But Classify and Predict give you the outputs, so you can take this and apply a custom activation function. Or do you want to change the loss function used for training? – Sebastian Nov 5 '14 at 0:56
• So, it is indeed, as I assumed, a specific "HiddenLayers" format that is needed, but I only tried to specify functions instead of names as strings :-( – Sjoerd C. de Vries Nov 5 '14 at 11:26
• Setting "L2Regularization" seems to hang the kernel in 10.4, has the option been changed? – Silvia Mar 17 '16 at 18:16

At least

Method -> {"NeuralNetwork", "L2Regularization" -> 0.01,  "HiddenLayers" -> {4, 3, 3}}


seems to work. "L1Regularization" doesn't seem to be settable. Haven't found out about activation functions yet.

Update

More options can be found using

Options@MachineLearningPackageScopeNeuralNetworkPredictor


{"BinaryEncoder" -> "Identity", "ClassNumber" -> Automatic, "CrossValidation" -> Automatic, "DataSize" -> All, "EarlyStopping" -> Automatic, "HiddenLayers" -> Automatic, "L2Regularization" -> Automatic, "LearningTask" -> "Predict", MaxIterations -> Automatic, NominalVariables -> Automatic, PerformanceGoal -> Automatic, "PrintCost" -> False, "OptimizationMethod" -> Automatic, "WeightInitializationMethod" -> Automatic}

• Thanks, this is a great start. I would really like to be able to set the activation functions though, so if you can figure that out, this will be the accepted answer. – mdeceglie Nov 4 '14 at 16:02
• @Italianice There really doesn't seem to be a named option to specify activation functions, but I have reasons to believe that it should be possible to do that using a particular format for the "HiddenLayers" option specification. I couldn't find out what would do the trick though. – Sjoerd C. de Vries Nov 4 '14 at 21:24
• Does running Options[model] provide any clues? The output contains "NeuronTypes" -> {"Tanh", "Tanh", "Linear"}, but it is inside some nested association structure that I don't yet understand. – mdeceglie Nov 4 '14 at 22:41
• Do you know how can I change "CrossValidation"? – Miguel May 26 '15 at 11:50
• @Miguel You might want to look into MachineLearningPackageScopeParseCrossValidationOptions . It seems to list its options, but I can't get it to work. – Sjoerd C. de Vries May 26 '15 at 20:55