class nbla::Lars
-
template<typename T>
class Lars : public nbla::Solver LARS solver defined as.
\[\begin{split} \lambda \leftarrow \eta \frac{\| w_t \|}{\| \Delta w_t + \beta w_t \|} \\ v_{t+1} \leftarrow m v_t + \gamma \lambda (\Delta w_t + \beta w_t) \\ w_{t+1} \leftarrow w_t - v_{t+1} \end{split}\]See also
See the paper linked below for more details. Yang Youity, Igor Gitmann, and Boris Ginsburg: Large Batch Training of Convolutional Networks https://arxiv.org/pdf/1708.03888
- Param lr:
\(\gamma\) Learning rate.
- Param momentum:
\(m\) Momentum.
- Param coefficient:
\(\eta\) Coefficient of the local learning rate.
- Param eps:
\(\epsilon\) Tiny factor for avoiding 0-division.
Public Functions
-
inline virtual float learning_rate()
Set learning rate.