Why Move Against the Gradient

Now we put the Taylor series to work. The question: standing at parameters $\theta = (w, b)$, which small move lowers the loss the most?

The move

Write the parameters as a vector $\theta = (w, b)$ and propose a change $\eta\,u$:

Here $u$ is a direction (a vector of length 1) and $\eta$ is a small positive number that controls the step size. We keep $\eta$ small because the Taylor approximation only holds close by.

We do not yet know which $u$ to use. The Taylor series will tell us.

Gradients in one minute

A function of two variables has a slope in each direction.

For $L(w, b) = w^2 + b^2$:

These are partial derivatives: the derivative with respect to one variable while treating the other as a constant. (That is why the $b^2$ term disappears when differentiating with respect to $w$.)

Collect them into a vector and you get the gradient:

The formula becomes a vector of numbers when you evaluate it at a point. At $(w, b) = (1, -2)$ the gradient is $(2, -4)$.

What Taylor says about the move

Apply the linear Taylor approximation to the new point:

The terms with $\eta^2$, $\eta^3$ and higher are far smaller when $\eta$ is small, so we drop them.

A move is good only if the loss goes down:

(We can ignore $\eta$ here because it is positive.) So any $u$ with $u^{T}\nabla L < 0$ is an improvement. Which one is best?

The best direction

The dot product of two vectors is

where $\beta$ is the angle between $u$ and the gradient. With $\|u\| = 1$, the length $\|\nabla L\|$ is fixed by where we are standing, so all we can change is $\cos\beta$, which lies between $-1$ and $1$.

So $u^{T}\nabla L$ is bounded:

It is most negative when $\cos\beta = -1$, that is, when $\beta = 180^\circ$. So the best $u$ points in exactly the opposite direction to the gradient.

That is where the famous phrase comes from: move in the direction opposite to the gradient. The gradient points uphill, where the loss rises fastest, so the opposite direction is the steepest way downhill.

The update rule

Folding the vector length into $\eta$, the rule is

or, written out for our two parameters:

The bar means "evaluated at the current values." The derivative formula is turned into a number by plugging in the current $w_t$ and $b_t$.

The whole algorithm is a short loop:

1. Start with random $w$ and $b$.
2. Repeat for a fixed number of iterations (or until the loss stops improving):
   1. Compute $\partial L / \partial w$ and $\partial L / \partial b$ at the current point.
   2. Update $w$ and $b$ using the rule above.

The update is gradient descent. The quantity $\eta$ is the learning rate.

Watching it go wrong: the learning rate

Take the simplest loss, $L(w) = w^2$. Its derivative is $2w$, so the update is

Start at $w = 1$ and see what different learning rates do:

The guarantee from the Taylor series holds only for a small step. Too large a step jumps over the minimum and can make things worse.

Two more things to watch for:

• The starting point matters. A loss with several valleys sends you to whichever valley is downhill from where you start. A local minimum is not necessarily the lowest point.
• Direction is guaranteed, not distance. Gradient descent promises that a small enough step reduces the loss. It does not promise the best possible final answer.