class nbla::Lion
-
template<typename T>
class Lion : public nbla::Solver LION solver defined as.
\[\begin{split} u = \beta_1 m_{t} + (1 - \beta_1) g_t\\ u = {\rm sign}(u)\\ m_{t+1} &\leftarrow \beta_2 m_{t} + (1 - \beta_2) g_t\\ w_{t+1} &\leftarrow w_t - \alpha \left( u + \lambda w_t \right) \end{split}\]See also
See the paper linked below for more details. Xiangning Chen et al., Symbolic Discovery of Optimization Algorithms. https://arxiv.org/abs/2302.06675
- Param lr:
\(\alpha\) Learning rate.
- Param beta1:
\(\beta_1\) Decay rate.
- Param beta2:
\(\beta_2\) Decay rate.
Public Functions
-
inline virtual float learning_rate()
Set learning rate.