Momentum
/moʊˈmɛntəm/
An optimization technique that accelerates gradient descent by accumulating a velocity vector in directions of persistent reduction in the loss.
In italiano: MomentumMomentum helps navigate ravines and accelerates convergence. It adds a fraction of the previous update vector to the current one, smoothing optimization trajectory and reducing oscillations.
Examples
- SGD with momentum 0.9
- Nesterov momentum
- Accelerating convergence