# What are the default parameters for ADAM optimizer?

ADAM Optimizer (used for optimizing the loss function for neural networks) consists of 3 free parameters:

• $\eta$, stepsize/learning rate
• $\beta_1$, forgetting factor for gradients
• $\beta_2$, forgetting factor for second moments of gradients
• $\epsilon$, smallest number for preventing division by $0$.

From the Mathematica documentation, $\beta_1 = 0.9$ and $\beta_2 = 0.999$. But what are the values for $\eta$ and $\epsilon$?
• I think this is how you change these parameters since it doesn't give an error during training: Method -> { "ADAM", "LearningRate" -> 0.001, "Beta1" -> 0.9, "Beta2" -> 0.999, "Epsilon" -> Rational[1, 100000]} – Miladiouss Jan 25 '18 at 4:16