Regularization: weight decay and dropout

The concept

Large networks have enough capacity to memorize their training data, which leads to overfitting. Regularization is any technique that reduces this by constraining the model. Besides early stopping and data augmentation, two classic methods are the L2 penalty, also called weight decay, and dropout.

L2 regularization adds λ·Σw² to the loss. It pulls the weights toward zero, preferring smaller weights and smoother functions. For one weight with squared error the penalized solution has the closed form w = Σxy / (Σx² + nλ), where n is the number of examples. In the example the data lie exactly on y = 2x. With λ = 0 gradient descent recovers w = 2. With λ = 1 the weight is shrunk to 1.765 and gradient descent agrees with the closed form. The penalty trades a bit of training accuracy for lower variance.

Dropout (Srivastava et al., 2014) randomly sets each unit's output to zero during training, with probability 1 − keep. In the inverted form used by modern libraries, the surviving activations are divided by keep during training so that their expected value stays the same, and no change is needed at test time, when all units are used. The program checks this by averaging over all 16 possible masks of four units, each equally likely when keep = 0.5. The average equals the original activations exactly. Dropout discourages units from relying on specific partners.

Code example

from itertools import product

activations = [0.2, 0.8, 0.5, 1.0]
keep = 0.5
masks = list(product([0, 1], repeat=len(activations)))
mean = [sum(a * mask[i] / keep for mask in masks) / len(masks) for i, a in enumerate(activations)]
print([round(value, 10) for value in mean] == activations, len(masks))

xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8]


def fit(penalty):
    w = 0.0
    n = len(xs)
    for _ in range(5000):
        grad = sum(2 * (w * x - y) * x for x, y in zip(xs, ys)) / n + 2 * penalty * w
        w -= 0.02 * grad
    return w


print(round(fit(0.0), 3), round(fit(1.0), 3))
closed = sum(x * y for x, y in zip(xs, ys)) / (sum(x * x for x in xs) + len(xs) * 1.0)
print(round(closed, 3))

True 16
2.0 1.765
1.765

Follow the code

Open the walkthrough to inspect the example one step at a time.

—

—

True 16

True 16

True 16
2.0 1.765

True 16
2.0 1.765
1.765

Behind the scenes

Cost and common mistakes

The L2 term adds O(1) work per weight per step. Dropout adds O(units) to create and apply the mask.

• Too much regularization causes underfitting: the weights are squeezed until the model can no longer fit the signal.
• Dropout is applied only during training. At test time use every unit, with no random masking.
• The penalty strength λ is a hyperparameter. Choose it with validation data, not the training set.

Try it yourself

What does the closed form give for λ = 0?

Further reading

• Srivastava et al. (2014): Dropout: A Simple Way to Prevent Neural Networks from Overfitting (external)
• Deep Learning (Goodfellow, Bengio, Courville) (external)
• Dive into Deep Learning (external)