Feedforward Networks: Notation and the Forward Pass

Backpropagation is easiest to follow once the network's structure and notation are fixed. We will use them for the rest of the module.

Structure

A feedforward network passes information in one direction, from the input to the output, with no loops. It has:

• an input layer, the vector $x$,
• $L - 1$ hidden layers,
• an output layer.

There are $L$ layers of weights. Each layer does the same two-step job.

Step 1, the weighted sum (called the pre-activation):

Step 2, the activation function $g$ applied to each entry:

We start from $h_0 = x$. The last layer is different: it applies the output function $O$ instead of $g$:

The output function is softmax for classification and linear for regression, as in the previous module. Finally the loss $L(\hat y, y)$ compares the prediction with the true answer.

Shapes

If layer $k$ has $n_k$ neurons, then

• $h_k$, $a_k$ and $b_k$ are vectors of length $n_k$,
• $W_k$ is an $n_k \times n_{k-1}$ matrix: one row for each neuron in layer $k$, one column for each neuron in layer $k-1$.

Row $i$ of $W_k$ holds the weights going into neuron $i$ of layer $k$. The individual entries are $W_{k,ij}$: from neuron $j$ in layer $k-1$ to neuron $i$ in layer $k$.

Activation functions

The hidden activation $g$ is applied to each entry of the vector separately. Two classic choices:

• Sigmoid: $g(a) = \dfrac{1}{1 + e^{-a}}$, with outputs between 0 and 1.
• Tanh: $g(a) = \dfrac{e^{a} - e^{-a}}{e^{a} + e^{-a}}$, with outputs between $-1$ and 1.

The derivatives we will need later are tidy: for the sigmoid $g'(a) = g(a)\bigl(1 - g(a)\bigr)$ and for tanh $g'(a) = 1 - g(a)^2$. A third choice, ReLU, $g(a) = \max(0, a)$, is very common in modern networks, and we will use it later in the course.

Why "deep"?

Substitute the layers into each other and you see a nested function:

A deep network is one with many hidden layers, so this expression is a long composition. The long composition is what makes the derivatives interesting.

A worked forward pass

Take a network with 2 inputs, one hidden layer of 2 sigmoid neurons, and 2 softmax outputs. The parameters are

and the input is $x = (1, 2)$, with true class 2.

Layer 1. $a_1 = b_1 + W_1 x = (0.1 + 0.2 - 0.6,\; -0.1 + 0.4 + 0.2) = (-0.3,\; 0.5)$. $h_1 = \sigma(a_1) = (0.4256,\; 0.6225)$.

Layer 2. $a_2 = b_2 + W_2 h_1 = (0.5(0.4256) - 0.4(0.6225),\; -0.3(0.4256) + 0.8(0.6225) + 0.1) = (-0.0362,\; 0.4703)$.

Output. Softmax gives $\hat y = (0.376,\; 0.624)$.

Loss. The true class is 2, so the cross-entropy loss is $-\log 0.624 = 0.4716$.

We will reuse this exact example to see every backward-pass formula produce real numbers.

In code

import numpy as np

def sigmoid(a):
    return 1 / (1 + np.exp(-a))

def softmax(a):
    e = np.exp(a - a.max())        # subtracting the max avoids overflow
    return e / e.sum()

def forward(x, Ws, bs):
    """Returns the pre-activations a, the activations h (h[0] = x) and y_hat."""
    hs, as_ = [x], []
    h = x
    for k in range(len(Ws) - 1):           # hidden layers
        a = bs[k] + Ws[k] @ h
        h = sigmoid(a)
        as_.append(a)
        hs.append(h)
    a = bs[-1] + Ws[-1] @ h                # output layer
    as_.append(a)
    return as_, hs, softmax(a)

Run on the example above this gives $\hat y = (0.376, 0.624)$, as computed by hand. The function stores every $a_k$ and $h_k$ along the way, because the backward pass will need them.