Backpropagation answered "how do we train multi-layer networks?". In 1989 came an answer to a different question: "is it worth it?"
The theorem
George Cybenko (1989), and Kurt Hornik with colleagues (1989), proved a result now called the universal approximation theorem. In plain words:
A network with just one hidden layer and enough neurons can approximate any reasonable continuous function as closely as you like.
Nothing assumes the function is a simple logic gate. It could be a messy real-world relationship between inputs and outputs that nobody has a formula for.
The picture: towers and bars
The idea behind the proof is easy to see. Take an unknown curve. Build it from many thin rectangular "towers", each one produced by a small group of neurons. Each tower has a width and a height that the weights control. Few towers give a coarse, blocky fit. Many thin towers trace the curve closely.
Build a surface out of towers and see how adding more of them improves the match.
The proof needs smooth activations to make the sharp edges adjustable. See why a smooth curve is easier to learn with than a hard step.
Why it was exciting
Lesson 3 said a single neuron cannot even do XOR. This result went the other way: with a hidden layer, there is no function a network cannot approximate in principle. XOR is easy, and so are far harder problems.
What it does not promise
It is easy to read too much into the theorem. It says a good network exists. It does not say:
- how many neurons you will need (it can be a huge number),
- how to find the weights, which is the training problem, or
- that the network will generalise to new data and not just memorise examples.
"A solution exists" and "gradient descent will find it from my data" are very different claims. The next lesson is about the gap between them.