Sigmoid Towers in One Dimension

For perceptrons we asked what a network can represent. Now the same question for sigmoid neurons, this time with real numbers and approximate answers: given an unknown function $f(x)$, can a network of sigmoids produce output close to $f(x)$?

A sigmoid can act like a step

Recall from the first lesson of this module that for a single input $\sigma(wx + b)$ is centred where $wx + b = 0$. Increasing $w$ makes the S steeper. For a very large $w$ it is almost vertical:

So a sigmoid neuron with weight $k$ and bias $-ka$ is a step at the position $x = a$. We can place the step wherever we like by choosing $b$, and make its edge as sharp as we like by choosing $w$.

Two steps make a tower

Take two such steps, one at $a$ and one at $c$, with $a < c$, and subtract them:

• For $x < a$: both steps are 0, so the difference is 0.
• For $a < x < c$: the first step is 1 and the second is still 0, so the difference is 1.
• For $x > c$: both are 1, so the difference is 0 again.

The result is a flat-topped tower of height 1 between $a$ and $c$, zero elsewhere. Multiplying it by a number $h$ makes the tower height $h$.

Many towers trace any curve

Now take a target function $f(x)$ that we want to approximate. Cut the $x$ axis into small intervals. Over each interval build a tower as above, and set its height to the value of $f$ at the middle of the interval. Add all the towers.

The sum is a staircase that follows $f$. The thinner the towers, the closer the staircase hugs the curve, so we can approximate $f$ to any desired accuracy by using enough of them.

The network that does this is small in structure:

• Input: $x$.
• Hidden layer: two sigmoid neurons per tower, one for each edge.
• Output: a neuron that adds them, with weight $+h_i$ on the left edge of tower $i$ and $-h_i$ on the right edge.

The heights are the output weights, and the positions of the edges are set by the hidden biases.

What we have shown, and what we have not

This construction is the core of the universal approximation theorem: for any reasonable function there is a sigmoid network that approximates it as closely as you like, provided it is allowed enough neurons.

Be careful about what it says:

• It is about existence. It tells us a good network exists, and how to build one in principle. It does not say gradient descent will find it.
• It is expensive. Ten towers cost 20 hidden neurons. To double the resolution you double the towers. The networks we use in practice are far more economical, and later lessons explain how.
• The edges are not perfect. With finite $w$ the step has a slope, so each tower has slightly sloped sides. More sharpness means larger weights.