2
$\begingroup$

ADAM Optimizer (used for optimizing the loss function for neural networks) consists of 3 free parameters:

  • $\eta$, stepsize/learning rate
  • $\beta_1$, forgetting factor for gradients
  • $\beta_2$, forgetting factor for second moments of gradients
  • $\epsilon$, smallest number for preventing division by $0$.

See Wikipedia for more information.

From the Mathematica documentation, $\beta_1 = 0.9$ and $\beta_2 = 0.999$. But what are the values for $\eta$ and $\epsilon$?

$\endgroup$
  • $\begingroup$ Someone has tried to answer this question here, but their code is buggy and doesn't show what eta is. $\endgroup$ – Miladiouss Jan 25 '18 at 3:56
  • $\begingroup$ I think this is how you change these parameters since it doesn't give an error during training: Method -> { "ADAM", "LearningRate" -> 0.001, "Beta1" -> 0.9, "Beta2" -> 0.999, "Epsilon" -> Rational[1, 100000]} $\endgroup$ – Miladiouss Jan 25 '18 at 4:16
  • $\begingroup$ I also found this Wolfram tutorial where under "Training Algorithm" they talk about how a default learning rate is chosen. However, when I manually choose a learning rate based on their formula (eta = (max(x) - min(x))/2), I CLEARLY get deferent results (i.e. little to nothing training). $\endgroup$ – Miladiouss Jan 26 '18 at 3:17
  • $\begingroup$ Mathematica uses MxNet and the default values MxNet are: learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8 The come from this website $\endgroup$ – Miladiouss Jan 29 '18 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.