Evaluating classifiers: accuracy, precision, recall, F1

Build a confusion matrix and see why accuracy alone can be misleading for rare classes.

def confusion(actual, predicted):
    tp = sum(a == 1 and p == 1 for a, p in zip(actual, predicted))
    fp = sum(a == 0 and p == 1 for a, p in zip(actual, predicted))
    fn = sum(a == 1 and p == 0 for a, p in zip(actual, predicted))
    tn = sum(a == 0 and p == 0 for a, p in zip(actual, predicted))
    return tp, fp, fn, tn


def metrics(actual, predicted):
    tp, fp, fn, tn = confusion(actual, predicted)
    accuracy = (tp + tn) / len(actual)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    return accuracy, precision, recall, f1


actual = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
predicted = [1, 1, 0, 0, 0, 0, 0, 0, 1, 0]
tp, fp, fn, tn = confusion(actual, predicted)
print(f"tp={tp} fp={fp} fn={fn} tn={tn}")
print(*[round(value, 3) for value in metrics(actual, predicted)])

rare = [1] * 10 + [0] * 990
always_negative = [0] * 1000
accuracy, precision, recall, f1 = metrics(rare, always_negative)
print(accuracy, recall)

tp=2 fp=1 fn=2 tn=5
0.7 0.667 0.5 0.571
0.99 0.0

Explanation

Machine learning builds models from data, so judging a model is as important as building it. For a binary classifier the confusion matrix counts four outcomes: true positives (predicted positive and actually positive), false positives, false negatives, and true negatives. Every common metric is computed from these four counts.

Accuracy is the fraction of correct predictions. Precision, tp / (tp + fp), answers: when the model says positive, how often is it right? Recall, tp / (tp + fn), answers: of all the real positives, how many did the model find? The F1 score is the harmonic mean of precision and recall, which is high only when both are high. Which metric matters depends on the cost of each mistake: a spam filter wants high precision, while a medical screening test usually wants high recall.

Accuracy can be badly misleading when classes are imbalanced. The last example has 10 positives among 1000 cases. A useless model that always predicts negative is 99 percent accurate and has recall 0, because it finds none of the cases that matter. Evaluate on data the model did not train on, and for rare events look at precision and recall, not accuracy alone.

—

tp=2 fp=1 fn=2 tn=5

tp=2 fp=1 fn=2 tn=5
0.7 0.667 0.5 0.571

tp=2 fp=1 fn=2 tn=5
0.7 0.667 0.5 0.571
0.99 0.0

Cost and common mistakes

Computing the confusion matrix is O(n) for n predictions.

• Accuracy hides failure on rare classes. Report precision and recall when classes are imbalanced.
• Evaluating on the training data overstates performance. Keep a separate test set for the final estimate.
• Precision and recall trade off against each other. Changing the decision threshold moves you along that trade-off.

Try it yourself

A screening test misses 20 of 100 sick patients and falsely flags 30 of 900 healthy ones. What is the recall?

Further reading

• scikit-learn user guide (external)
• An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani) (external)
• Stanford CS229: Machine Learning (external)