NetTrain automatic settings

I'm trying to compare Mathematica's NetTrain with other algorithms, and I used Automatic settings as they were working quite well.

Now I need to include those results in a paper, but I don't know which method mathematica choose for training.

There are three of those - ADAM, RMSProp and SGD, by altering between those and comparing performance with automatic I found out that ADAM seems to be the one used for my case. But I still don't know a lot of parameters like "LearningRate", "LearningRateSchedule", etc. Is there a way to find them out?

• Sometimes you can get at these with InternalInheritedBlock. Do you have a sample net to work with? The settings may change. In any case you can always look through the DownValues with GeneralUtilitiesPrintDefinitionsLocal. – b3m2a1 Aug 14 '17 at 7:56

For now, you can use the "cheat code" NeuralNetworksPrivateMXTrainer$OptimizerSpec, which will show the current defaults for the different methods (some internal details are mixed in here): In[32]:= NeuralNetworksPrivateMXTrainer$OptimizerSpec/.NeuralNetworksDefaulting[_,d_]:>d//GeneralUtilitiesPrettyForm
Out[32]= <|
<|Beta1 -> 0.9, Beta2 -> 0.999, Epsilon -> Rational[1, 100000]|>
},
SGD -> {
{SGD, #}&,
<|Momentum -> 0.93, LearningRateSchedule -> Polynomial|>
},
RMSProp -> {
{RMSProp, #}&,
<|
Beta -> 0.95, Momentum -> 0.9, Epsilon -> Rational[1, 100000000]
|>
},
$CommonSuboptions -> <| L2Regularization -> ArrayCasesT[Nullable[TensorT[{}, RealT]], 0.], GradientClipping -> ArrayCasesT[Nullable[TensorT[{}, RealT]], None], WeightClipping -> ArrayCasesT[Nullable[TensorT[{}, RealT]], None], InitialLearningRate -> Automatic, LearningRateSchedule -> None, GradientsBag -> None, GradientMapper -> RMSEnergy |> |>  The rule is currently that SGD is used for networks with less than 128 parameters (total weight components), otherwise ADAM is used. • what the default learning rate of each methods? – partida Dec 10 '17 at 3:15 • For the ADAM case, I can see that$beta1 = 0.9$,$beta2 = 0.999$,$epsilon = 1e-5$. But what is the main parameter$alpha\$? – Miladiouss Jan 25 '18 at 1:18
• This doesn't work on Mathematica 11.3. – Miladiouss Jan 25 '18 at 1:28