Recap: Derivatives and Gradient Descent in Networks

A short reference for what this module and the previous one built.

The vocabulary of derivatives

Term	What it is	Example
Derivative	slope of a function of one variable	$\dfrac{d}{dw} w^2 = 2w$
Partial derivative	slope with respect to one variable, others held constant	$\dfrac{\partial}{\partial w}(w^2 + b^2) = 2w$
Gradient	vector of all partial derivatives	$\nabla L = (2w, 2b)$
Hessian	matrix of second partial derivatives	needed for second-order Taylor terms
Chain rule	derivative of a composite function is a product of derivatives	the heart of backpropagation

A derivative evaluated at a point is a number. A formula for the derivative becomes a number only once you substitute the current parameter values.

Gradient descent

Choose a loss that measures how wrong the model is.
Start with random parameters $\theta$ .
Repeat: compute $\nabla L(\theta)$ and update $\theta \leftarrow \theta - \eta\,\nabla L(\theta)$ .

The reason the minus sign works: the Taylor series says a small step in direction $u$ changes the loss by about $\eta\,u^T\nabla L$ , and the most negative value comes from $u$ pointing opposite to the gradient.

The four backpropagation formulas

For softmax with cross-entropy and a feedforward network with activation $g$ :

Step	Formula
Output gradient	$\delta_L = \hat y - e_\ell$
Weights and biases of layer $k$	$\nabla_{W_k}L = \delta_k\,h_{k-1}^{T}, \quad \nabla_{b_k}L = \delta_k$
Through the weights	$\dfrac{\partial L}{\partial h_{k-1}} = W_k^{T}\delta_k$
Through the activation	$\delta_{k-1} = \dfrac{\partial L}{\partial h_{k-1}} \odot g'(a_{k-1})$

Checklist when training misbehaves

Loss goes up or explodes. The learning rate is probably too large. Try a smaller $\eta$ .
Loss barely moves. It may be on a plateau (saturated sigmoids), or $\eta$ may be too small. Early layers with sigmoid activations can be very slow because of vanishing gradients.
Loss refuses to fall and you suspect a bug. Run a gradient check against numerical gradients on a tiny network.
Shapes do not match. Every gradient should have the same shape as the parameter it belongs to.
All neurons behave the same. Check that the weights were initialised randomly.

Where this leaves us

We now have, for a network of any depth: a forward pass, a way to get every gradient, and gradient descent. What remains are ways to make training faster and more reliable, and architectures built for particular kinds of data.

MediumRecap

Which part of backpropagation uses the chain rule, and which part uses the outer product?

EasyRecap