Lessons from the History

The whole timeline in one picture, the patterns that keep repeating, and where to go next.

A Brief History of Deep Learning

Timeline from 1943 to 2022 showing the spring of AI, the winter, and the deep revival, with key events marked
Spring, winter, revival. Dates are spaced for readability, not to scale.

Four patterns worth remembering

  1. Ideas wait for their moment. Backpropagation, convolutional networks and attention-like ideas existed long before they changed the world. They needed data, compute or a training trick to catch up.
  2. Progress comes from other fields. Microscopes settled the neuron debate. Graphics cards for video games made deep learning fast. Astronomers gave us gradient descent.
  3. Hype and disappointment repeat. Bold predictions in 1958 were followed by a winter. Be interested in claims, but look for the benchmark and the evidence.
  4. Simple ideas scale. A weighted sum, a threshold and a learning rule in 1958. A weighted sum, a nonlinearity and gradient descent in today's models. Most of the gain came from doing the same simple thing with far more data and compute.

Check yourself

EasyTimeline

Order these events: AlexNet, Perceptron, Transformer, Backpropagation popularised, Minsky and Papert's book.

MediumHistory

Why did a technique that existed in 1986 only transform the field around 2012?

HardCritical thinking

A startup claims its new model will soon replace all human experts. Using this history, how would you evaluate the claim?

Where next

You now have the map. To build the machinery itself, continue with the Deep Learning course and its simulator, and try the Deep Learning simulator directly.

For deeper reading on this history, Jürgen Schmidhuber's survey "Deep Learning in Neural Networks: An Overview" (2015) and the 2015 Nature review "Deep learning" by LeCun, Bengio and Hinton are good places to start.