Sending the Gradient Back Through a Hidden Layer

How the gradient at one layer's pre-activation turns into the gradient at the layer before it: multiply by the transposed weights, then by the activation's slope.

We can now compute the weight and bias gradients of layer $k$ once we know $\delta_k$ . The remaining question is how to get $\delta_{k-1}$ , the same quantity for the layer before, from $\delta_k$ . There are two links to cross: from $a_k$ back to $h_{k-1}$ , and from $h_{k-1}$ back to $a_{k-1}$ .

Link 1: back through the weights

The activation $h_{k-1,j}$ feeds every neuron of layer $k$ , through the weights $W_{k,ij}$ for each $i$ :

a_{k,i} = b_{k,i} + \sum_j W_{k,ij}\,h_{k-1,j}

So a change in $h_{k-1,j}$ changes all the $a_{k,i}$ at once, and the loss feels the sum of those effects. The chain rule now has one term for each $i$ :

\frac{\partial L}{\partial h_{k-1,j}} = \sum_i \frac{\partial L}{\partial a_{k,i}}\,\frac{\partial a_{k,i}}{\partial h_{k-1,j}} = \sum_i \delta_{k,i}\,W_{k,ij}

In matrix form this sum is a matrix-vector product with the transposed weights:

\boxed{\;\frac{\partial L}{\partial h_{k-1}} = W_k^{T}\,\delta_k\;}

The intuition: the error travels back along the same connections it came through, each scaled by the weight on that connection. A neuron that contributed strongly to a large error receives a large share.

Link 2: back through the activation

The activation is applied to each entry on its own: $h_{k-1,j} = g(a_{k-1,j})$ . Changing $a_{k-1,j}$ affects only $h_{k-1,j}$ , so there is a single term:

\delta_{k-1,j} = \frac{\partial L}{\partial a_{k-1,j}} = \frac{\partial L}{\partial h_{k-1,j}}\; g'(a_{k-1,j})

In vector form, with $\odot$ for element-by-element multiplication:

\boxed{\;\delta_{k-1} = \bigl(W_k^{T}\,\delta_k\bigr) \odot g'(a_{k-1})\;}

The slope $g'$ depends on the activation function:

Activation	$g'$ in terms of the stored values
Sigmoid	$h\,(1 - h)$
Tanh	$1 - h^2$
ReLU	1 if $a > 0$ , otherwise 0

Because the forward pass stored $h_{k-1}$ , we get $g'$ for free.

Diagram showing delta at layer k feeding the weight gradient, the bias gradient, and the gradient to the previous hidden layer — From delta at layer k: read off the weight and bias gradients, then pass the error one layer back.

Continuing the worked example

For the two-layer network we had $\delta_2 = (0.376, -0.376)$ and

W_2 = \begin{pmatrix} 0.5 & -0.4 \\ -0.3 & 0.8 \end{pmatrix},\qquad h_1 = (0.4256,\; 0.6225)

Link 1:

W_2^{T}\delta_2 = \begin{pmatrix} 0.5 & -0.3 \\ -0.4 & 0.8 \end{pmatrix}\begin{pmatrix} 0.376 \\ -0.376 \end{pmatrix} = \begin{pmatrix} 0.3008 \\ -0.4512 \end{pmatrix}

Link 2: the sigmoid slope is $h_1(1 - h_1) = (0.2445,\; 0.2350)$ , so

\delta_1 = (0.3008 \times 0.2445,\; -0.4512 \times 0.2350) = (0.0735,\; -0.1060)

Then the first layer's gradients follow from the previous lesson:

\nabla_{W_1} L = \delta_1\,x^{T} = \begin{pmatrix} 0.0735 & 0.1471 \\ -0.1060 & -0.2121 \end{pmatrix},\qquad \nabla_{b_1} L = \delta_1

A numerical check again agrees with every entry. The whole gradient of this network, all 12 parameters, came from a few matrix operations.

About the dimensions

The derivative of a vector with respect to another vector is a matrix, with one entry for each pair of components. Here that matrix is $W_k$ for the link through the weights, and a diagonal matrix of slopes for the element-wise activation. Multiplying by these matrices, or by their transposes going backwards, is what the two boxed formulas do.

EasyBackpropagation

Given W2 = [[1, 2], [0, -1]] and delta2 = (0.5, 1), compute W2 transposed times delta2.

MediumBackpropagation

Why does the backward pass use the transpose of the weight matrix?

MediumActivation

Link 1: back through the weights

Link 2: back through the activation

Continuing the worked example

Given W2 = [[1, 2], [0, -1]] and delta2 = (0.5, 1), compute W2 transposed times delta2.

Why does the backward pass use the transpose of the weight matrix?

Why is g-prime applied element by element and not through a matrix multiplication?