Momentum

Let the update build up speed by remembering past gradients: faster on flat ground, calmer across steep walls, with a risk of overshooting.

Plain gradient descent has no memory. Every step depends only on the gradient at the current point. On a flat region it takes tiny steps, and in a narrow valley it bounces between walls. Momentum adds memory.

The idea

Picture a heavy ball rolling down the surface. It does not stop and restart at each point. It carries velocity, which builds up when the slope keeps pointing the same way and is damped when the slope keeps changing sign.

We keep a velocity vector $v$ that accumulates past gradients:

v_t = \gamma\,v_{t-1} + \eta\,\nabla L(\theta_t), \qquad \theta_{t+1} = \theta_t - v_t

The number $\gamma$ , typically 0.9, says how much of the old velocity survives. With $\gamma = 0$ this is ordinary gradient descent.

Unrolling the recursion shows what $v$ is:

v_t = \eta\bigl(g_t + \gamma\,g_{t-1} + \gamma^2 g_{t-2} + \cdots\bigr)

It is an exponentially weighted sum of past gradients, recent ones counting most. Two effects follow:

If the gradient keeps the same sign, the terms add up. In the long run the step is about $\eta/(1 - \gamma)$ , which is $10\eta$ for $\gamma = 0.9$ .
If the gradient alternates in sign, as it does across a valley, the terms largely cancel, which damps the zigzag.

Where it helps: the flat start

The sigmoid toy problem starts on a plateau. With the same learning rate, the number of steps until the loss stays below $10^{-3}$ is:

Rate $\eta$	Plain GD	Momentum ( $\gamma = 0.9$ )
0.1	1815	177
0.3	606	124
1.0	183	148

With a small rate, momentum is about ten times faster: the velocity builds up across the flat stretch. As plain gradient descent gets a good rate the advantage shrinks, because it is no longer crawling.

Where it can hurt: overshoot

A ball with velocity does not stop at the bottom. It rolls through and climbs the other side. On the narrow valley with $\eta = 0.05$ :

Step	Momentum $(w, b)$	Plain GD $(w, b)$
0	(1, 1)	(1, 1)
2	(−0.20, 0.86)	(0.25, 0.90)
4	(−0.84, 0.58)	(0.06, 0.81)
6	(0.03, 0.25)	(0.02, 0.74)

Momentum makes much more progress along the flat direction: $b$ is 0.25 after six steps against 0.74. The price is that $w$ swings past zero and back, since the accumulated velocity carries it across. Overall on this small valley, momentum needs 102 steps to settle below $10^{-4}$ against 84 for plain gradient descent at the same rate. Momentum is not a guaranteed improvement. It helps most when the plain method is crawling, and it needs a rate and $\gamma$ that do not overshoot too much.

In code

def momentum_step(theta, v, grad, eta=0.05, gamma=0.9):
    v = gamma * v + eta * grad(theta)
    return theta - v, v

Start with $v = 0$ and call it repeatedly. The velocity is the only extra state.

Try it yourself

Optimizer playground →

Compare gradient descent and momentum on the plateau and in the narrow valley.

EasyMomentum

Write the momentum update and say what gamma equal to 0 gives.

MediumMomentum

Why does momentum damp oscillations across a narrow valley?