L2 Regularization and Weight Decay

The high-degree polynomial overfits because its coefficients can grow huge and bend the curve through every noisy point. Regularization means any change to learning that aims to reduce the test error, even at some cost in training error. The most common is L2 regularization.

The idea

Change the objective. Instead of minimising the data loss $L$ alone, minimise

The second term is a penalty on large weights, and $\lambda \ge 0$ controls its strength. Now there is a trade-off: a weight may only grow if it reduces the data loss by more than the penalty it costs. With $\lambda = 0$ we have the original problem. With $\lambda$ very large, all weights are pushed towards zero.

In practice the bias terms are usually left out of the penalty, and in these demonstrations we penalise all coefficients for simplicity.

For a linear model with squared error there is a closed form (called ridge regression):

The added $\lambda I$ also makes the matrix easier to invert, which is why the polynomial fits above are stable when $\lambda > 0$.

Weight decay

The gradient of the penalty $\lambda\lVert w\rVert^2$ is $2\lambda w$. So a gradient descent step on the regularised loss is

Before applying the usual gradient step, every weight is first multiplied by a number just below 1. With $\eta = 0.1$ and $\lambda = 0.01$ the factor is $0.998$. The weights decay towards zero unless the data gradient keeps them up, which is why L2 regularization in neural networks is also called weight decay.

Effect on the polynomial fit

Take the degree-12 polynomial, which overfits badly, and fit it with different $\lambda$:

import numpy as np
from numpy.polynomial import legendre

def true_f(x):
    return np.sin(2 * np.pi * x)

def features(x, degree):
    return legendre.legvander(2 * x - 1, degree)

def fit(x, y, degree, lam=0.0):
    A = features(x, degree)
    return np.linalg.solve(A.T @ A + (lam + 1e-9) * np.eye(degree + 1), A.T @ y)

x_train = np.linspace(0, 1, 15)
x_test = np.linspace(0.005, 0.995, 200)
noise = 0.2
rng = np.random.default_rng(2)

for lam in (0, 0.001, 0.01, 0.1, 1, 10):
    preds, norms = [], []
    for _ in range(1000):
        y = true_f(x_train) + rng.normal(0, noise, 15)
        w = fit(x_train, y, 12, lam)
        preds.append(features(x_test, 12) @ w)
        norms.append(np.linalg.norm(w))
    preds = np.array(preds)
    bias_sq = np.mean((preds.mean(axis=0) - true_f(x_test)) ** 2)
    variance = np.mean(preds.var(axis=0))
    print(f"lambda {lam:6}: bias^2 {bias_sq:.4f}  variance {variance:.4f}  test error {bias_sq + variance + noise**2:.4f}  typical |w| {np.median(norms):.2f}")

Output:

lambda      0: bias^2 0.0002  variance 0.2237  test error 0.2639  typical |w| 2.01
lambda  0.001: bias^2 0.0000  variance 0.1993  test error 0.2394  typical |w| 1.97
lambda   0.01: bias^2 0.0003  variance 0.1255  test error 0.1658  typical |w| 1.85
lambda    0.1: bias^2 0.0012  variance 0.0356  test error 0.0768  typical |w| 1.58
lambda      1: bias^2 0.0329  variance 0.0107  test error 0.0836  typical |w| 1.15
lambda     10: bias^2 0.2799  variance 0.0016  test error 0.3215  typical |w| 0.37

This is the bias-variance trade-off made visible:

• Raising $\lambda$ from 0 to 0.1 cuts the variance from 0.22 to 0.036 at almost no cost in bias (0.0012). The test error falls from 0.264 to 0.077, back near the best polynomial of lower degree (0.054) even though the model still has 13 coefficients.
• Beyond that, bias grows. At $\lambda = 10$ the weights are squeezed so hard that the model underfits (bias squared 0.28) and the test error is worse than with no regularization at all.

So $\lambda$ is a dial between overfitting and underfitting. Choose it with a validation set, not the test set.

A note on L1

Replacing the squared penalty by the absolute value, $\lambda\sum_i |w_i|$, gives L1 regularization. Its effect is different: it tends to push many weights to exactly zero, producing sparse models. L2 shrinks all weights smoothly but rarely makes any exactly zero.