class nbla::Lion

template<typename T>
class Lion : public nbla::Solver

LION solver defined as.

\[\begin{split} u = \beta_1 m_{t} + (1 - \beta_1) g_t\\ u = {\rm sign}(u)\\ m_{t+1} &\leftarrow \beta_2 m_{t} + (1 - \beta_2) g_t\\ w_{t+1} &\leftarrow w_t - \alpha \left( u + \lambda w_t \right) \end{split}\]

See also

See the paper linked below for more details. Xiangning Chen et al., Symbolic Discovery of Optimization Algorithms. https://arxiv.org/abs/2302.06675

Param lr:

\(\alpha\) Learning rate.

Param beta1:

\(\beta_1\) Decay rate.

Param beta2:

\(\beta_2\) Decay rate.

Public Functions

inline virtual float learning_rate()

Set learning rate.