Linear regression with gradient descent
Regression predicts a number. Linear regression models the output as ŷ = w·x + b with a slope w and an intercept b. Training means finding the w and b that minimize a loss. The usual choice is the mean squared error (MSE), the average of (ŷ − y)² over the training examples.
Python
xs = [0, 1, 2, 3, 4]
ys = [1, 3, 5, 7, 9]
w = b = 0.0
n = len(xs)
for _ in range(2000):
grad_w = sum(2 * (w * x + b - y) * x for x, y in zip(xs, ys)) / n
grad_b = sum(2 * (w * x + b - y) for x, y in zip(xs, ys)) / n
w -= 0.05 * grad_w
b -= 0.05 * grad_b
mean_x = sum(xs) / n
mean_y = sum(ys) / n
closed_w = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys)) / sum((x - mean_x) ** 2 for x in xs)
closed_b = mean_y - closed_w * mean_x
print(round(w, 3), round(b, 3))
print(closed_w, closed_b)