The Gradient at the Output Layer

Backpropagation needs a place to start, the gradient of the loss with respect to the last pre-activation $a_L$. For classification we use softmax with cross-entropy, and the result is unusually clean.

Setup

Let $a = a_L$ be the vector of $K$ numbers before the output function. Then

where $\ell$ is the index of the true class. We want $\dfrac{\partial L}{\partial a_j}$ for every $j$. The chain passes through $\hat y$, so we proceed in two steps.

Step 1: the loss with respect to the prediction

The loss depends on $\hat y$ only through the entry $\hat y_\ell$, so

Step 2: the softmax derivative

Each output $\hat y_i$ depends on all the entries of $a$, because every $a_k$ appears in the denominator. Differentiating gives

where $\mathbb{1}[i = j]$ is 1 if $i = j$ and 0 otherwise. In words:

• for $i = j$: $\hat y_i(1 - \hat y_i)$,
• for $i \ne j$: $-\hat y_i\,\hat y_j$.

A quick derivation of the first case. Write $S = \sum_k e^{a_k}$, so $\hat y_i = e^{a_i}/S$. Then $\partial \hat y_i/\partial a_i = e^{a_i}/S - e^{a_i}e^{a_i}/S^2 = \hat y_i - \hat y_i^2$. For $i \ne j$ the numerator does not depend on $a_j$, so only the denominator contributes, giving $-e^{a_i}e^{a_j}/S^2 = -\hat y_i \hat y_j$.

Combine with the chain rule

The loss reaches $a_j$ through every $\hat y_i$:

Only the term $i = \ell$ survives, because every other $\partial L/\partial \hat y_i$ is 0:

The $\hat y_\ell$ factors cancel, leaving a very clean result:

where $e_\ell$ is the one-hot vector for the true class (1 in position $\ell$, 0 elsewhere). The gradient is simply the prediction minus the target.

What it means

• For a wrong class $j$, the gradient is $\hat y_j > 0$. Gradient descent subtracts it, pushing that class's score down, more strongly the more probability the network wrongly gave it.
• For the true class, the gradient is $\hat y_\ell - 1 < 0$. Subtracting it pushes the score up.
• The entries always sum to 0 (the predictions sum to 1 and so does the one-hot vector), so the update moves probability between classes and does not create it.

Example. For the scores $a = (-1, 1, 2, 3)$ we found $\hat y = (0.012, 0.089, 0.242, 0.657)$. With the true class being the third, the gradient is

The fourth class has the highest predicted probability but is wrong, so it receives the strongest push down. The true class is pushed up.

The worked network from earlier. There $\hat y = (0.376, 0.624)$ and the true class is 2, so $\nabla_{a_2} L = (0.376,\; 0.624 - 1) = (0.376,\; -0.376)$.

Regression gives the same shape

For regression with a linear output, $\hat y = a$, and squared error $L = \tfrac12\sum_j(\hat y_j - y_j)^2$, we get

It is again "prediction minus target". That is part of why softmax with cross-entropy and linear output with squared error are such natural pairings: the gradient at the output takes the same simple form in both.