class nbla::RMSprop
-
template<typename T>
class RMSprop : public nbla::Solver -
This is defined as
\[\begin{split} g_t \leftarrow \Delta w_t\\ v_t \leftarrow \gamma v_{t-1} + \left(1 - \gamma \right) g_t^2\\ w_{t+1} \leftarrow w_t - \eta \frac{g_t}{\sqrt{v_t} + \epsilon} \end{split}\]See also
See the paper linked below for more details. Geoff Hinton Lecture 6a : Overview of mini-batch gradient descent http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
- Param lr:
\(\eta\) Learning rate.
- Param decay:
\(\gamma\) Decay rate.
- Param eps:
\(\epsilon\) Tiny factor for avoiding 0-division.
Public Functions
-
inline virtual float learning_rate()
Set learning rate.