Regularization: weight decay and dropout
Large networks have enough capacity to memorize their training data, which leads to overfitting. Regularization is any technique that reduces this by constraining the model. Besides early stopping and data augmentation, two classic methods are the L2 penalty, also called weight decay, and dropout.
Python
from itertools import product
activations = [0.2, 0.8, 0.5, 1.0]
keep = 0.5
masks = list(product([0, 1], repeat=len(activations)))
mean = [sum(a * mask[i] / keep for mask in masks) / len(masks) for i, a in enumerate(activations)]
print([round(value, 10) for value in mean] == activations, len(masks))
xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8]
def fit(penalty):
w = 0.0
n = len(xs)
for _ in range(5000):
grad = sum(2 * (w * x - y) * x for x, y in zip(xs, ys)) / n + 2 * penalty * w
w -= 0.02 * grad
return w
print(round(fit(0.0), 3), round(fit(1.0), 3))
closed = sum(x * y for x, y in zip(xs, ys)) / (sum(x * x for x in xs) + len(xs) * 1.0)
print(round(closed, 3))