Gradient Descent for a Sigmoid Neuron

The update rule needs two numbers at each step, $\partial L/\partial w$ and $\partial L/\partial b$. We now derive them for our sigmoid neuron and then run the algorithm.

The loss for one example

The model is $f(x) = \sigma(wx + b)$. For a single training example $(x, y)$ the loss is

Derivative with respect to $w$

Use the chain rule. The derivative of a square is twice the thing times the derivative of the thing, and the $\tfrac12$ cancels the 2:

The true value $y$ is a constant, so it contributes nothing to the derivative. What remains is the derivative of the sigmoid. Write $z = wx + b$. The sigmoid is $\sigma(z) = (1 + e^{-z})^{-1}$, so

This is a neat property: the derivative of the sigmoid is expressed using the sigmoid's own output. With $f(x) = \sigma(z)$ and the chain rule factor $\partial z/\partial w = x$:

Putting it together:

Derivative with respect to $b$

The same steps apply, except $\partial z / \partial b = 1$, so the factor $x$ disappears:

Many examples

With $N$ examples the loss is a sum, and the derivative of a sum is the sum of the derivatives:

Notice how much is shared: the two formulas differ only by the final $x_i$.

In code

We train on the toy data from before, the points $(0.5, 0.2)$ and $(2.5, 0.9)$, starting from $w = b = -2$ with learning rate $\eta = 1$:

import numpy as np

X = np.array([0.5, 2.5])
Y = np.array([0.2, 0.9])

def f(x, w, b):
    return 1 / (1 + np.exp(-(w * x + b)))

def loss(w, b):
    return 0.5 * np.sum((f(X, w, b) - Y) ** 2)

def gradients(w, b):
    pred = f(X, w, b)
    common = (pred - Y) * pred * (1 - pred)   # shared by both derivatives
    return np.sum(common * X), np.sum(common)

w, b, eta = -2.0, -2.0, 1.0
for epoch in range(1, 1001):
    dw, db = gradients(w, b)
    w -= eta * dw
    b -= eta * db
    if epoch in (1, 10, 100, 500, 1000):
        print(epoch, round(w, 3), round(b, 3), round(loss(w, b), 5))

Output:

1 -1.995 -1.992 0.41573
10 -1.942 -1.92 0.41484
100 1.003 -1.051 0.01774
500 1.785 -2.272 0.0
1000 1.792 -2.282 0.0

Reading the run

• The loss only goes down. Unlike guesswork, no step takes us uphill.
• The start is slow. From $(-2, -2)$ the gradient is tiny, about $(-0.0055, -0.0077)$, because both sigmoids are saturated there. The loss barely moves in the first ten epochs (0.4157 to 0.4148). Then the path reaches steeper ground and the loss drops fast. By epoch 100 it is 0.018.
• It ends where we wanted. The final $w \approx 1.79$, $b \approx -2.28$ is almost exactly the parameter pair we reached by hand earlier, $(1.78, -2.27)$, and the loss is essentially 0.

Completing the cycle

The story now mirrors the perceptron's:

Next, as we did for the perceptron, we ask what this kind of neuron can represent.