Logistic regression for classification

The concept

Despite its name, logistic regression is a classification method. It computes a linear score z = w·x + b and passes it through the sigmoid function σ(z) = 1 / (1 + e⁻ᶻ), which maps any number to a value between 0 and 1 that is read as the probability of the positive class. A prediction is positive when that probability is at least 0.5, which happens exactly when z >= 0.

The model is trained by minimizing the log loss (binary cross-entropy), which punishes confident wrong predictions heavily. Its gradient has the same pleasant shape as in linear regression: ∂L/∂w = average of (σ(z) − y)·x. The decision boundary, where the probability is 0.5, is at x = −b / w for one feature.

The data below are separable: every example with x <= 3 is class 0 and every example with x >= 4 is class 1. The trained model classifies all six correctly and places the boundary near 3.47, in the gap between 3 and 4. The exact position within the gap depends on how long training runs. For separable data, the unregularized weights would grow forever as training continues, so practical implementations add a regularization penalty or stop early.

Code example

import math

xs = [1, 2, 3, 4, 5, 6]
labels = [0, 0, 0, 1, 1, 1]


def sigmoid(z):
    return 1 / (1 + math.exp(-z))


w = b = 0.0
for _ in range(3000):
    grad_w = sum((sigmoid(w * x + b) - y) * x for x, y in zip(xs, labels)) / len(xs)
    grad_b = sum((sigmoid(w * x + b) - y) for x, y in zip(xs, labels)) / len(xs)
    w -= 0.5 * grad_w
    b -= 0.5 * grad_b

predictions = [1 if sigmoid(w * x + b) >= 0.5 else 0 for x in xs]
print(predictions == labels)
print(round(sigmoid(w * 2 + b), 3), round(sigmoid(w * 3.5 + b), 3), round(sigmoid(w * 5 + b), 3))
print(round(-b / w, 2))

True
0.001 0.537 0.999
3.47

Follow the code

Open the walkthrough to inspect the example one step at a time.

—

—

True

True
0.001 0.537 0.999

True
0.001 0.537 0.999
3.47

Behind the scenes

Cost and common mistakes

Each training step costs O(n) for n examples (O(n·d) with d features), the same as linear regression.

• Logistic regression produces a probability, not a certainty. Use a threshold appropriate to the costs of errors, not always 0.5.
• On perfectly separable data the weights keep growing, making the probabilities extreme. Use regularization.
• The model is linear in its inputs, so a single logistic regression cannot separate classes that need a curved boundary without feature engineering.

Try it yourself

Roughly what probability does the model give for x = 3.47?

Further reading

• Stanford CS229: Machine Learning (external)
• An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani) (external)
• scikit-learn user guide (external)