ADAM Optimizer (used for optimizing the loss function for neural networks) consists of 3 free parameters:
- $\eta$, stepsize/learning rate
- $\beta_1$, forgetting factor for gradients
- $\beta_2$, forgetting factor for second moments of gradients
- $\epsilon$, smallest number for preventing division by $0$.
See Wikipedia for more information.
From the Mathematica documentation, $\beta_1 = 0.9$ and $\beta_2 = 0.999$. But what are the values for $\eta$ and $\epsilon$?