Finding Weights by Hand

Before we build a learning algorithm, let us do the learning by hand once. It shows exactly what the algorithm will have to do.

OR as four inequalities

For two inputs, the perceptron computes $w_0 + w_1 x_1 + w_2 x_2$. It should be below 0 when OR is 0, and at least 0 when OR is 1. Substituting each input:

We need numbers $w_0, w_1, w_2$ that satisfy all four at once.

One solution

Try $w_0 = -1$, $w_1 = 1.1$, $w_2 = 1.1$:

• $-1 < 0$ ✓
• $-1 + 1.1 = 0.1 \ge 0$ ✓
• $-1 + 1.1 = 0.1 \ge 0$ ✓
• $-1 + 2.2 = 1.2 \ge 0$ ✓

Geometrically, the boundary is $-1 + 1.1 x_1 + 1.1 x_2 = 0$. It cuts off the origin on the negative side and leaves the other three points on the positive side, as in the previous lesson.

Many solutions, and wrong ones

This is not the only answer. Changing $w_0$ slides the line up or down (it controls where the line crosses the axes), and changing $w_1$ and $w_2$ tilts it. Plenty of values work.

Plenty do not. Try $w_0 = -1$, $w_1 = w_2 = 0.5$. The input $(0, 1)$ gives $-1 + 0.5 = -0.5 < 0$, so the neuron outputs 0 when OR says 1. That line puts a point on the wrong side, and the perceptron makes an error.

So the job is: find a line that puts every 0-input on one side and every 1-input on the other.

The same idea for the MP neuron

The MP neuron had the same problem with only one unknown, $\theta$. Writing the four inequalities for it gives a single number to find. That was easy to do by eye. With three unknowns, and then with hundreds, it stops being easy.

Why we need an algorithm

Solving by hand works for two inputs and four rows. A real problem has many features and thousands of examples, so no one can write and solve the inequalities by hand. We want a procedure that starts with any weights and improves them until every example is on the right side. That is the perceptron learning algorithm.