C++: stable softmax and cross-entropy loss
Turn raw scores into probabilities, measure the loss, and avoid overflow.
#include <algorithm>
#include <cmath>
#include <iomanip>
#include <iostream>
#include <vector>
std::vector<double> softmax(const std::vector<double>& z) {
double largest = *std::max_element(z.begin(), z.end());
std::vector<double> p(z.size());
double total = 0.0;
for (std::size_t i = 0; i < z.size(); ++i) {
p[i] = std::exp(z[i] - largest);
total += p[i];
}
for (double& v : p) {
v /= total;
}
return p;
}
int main() {
std::cout << std::fixed << std::setprecision(3);
std::vector<double> p = softmax({2.0, 1.0, 0.1});
std::cout << p[0] << " " << p[1] << " " << p[2] << "\n";
std::cout << "loss if class 0 is true: " << -std::log(p[0]) << "\n";
std::cout << "loss if class 2 is true: " << -std::log(p[2]) << "\n";
std::vector<double> big = softmax({1000.0, 1001.0});
std::cout << big[0] << " " << big[1] << "\n";
}0.659 0.242 0.099 loss if class 0 is true: 0.417 loss if class 2 is true: 2.317 0.269 0.731
Explanation
A classifier network ends with a vector of raw scores called logits. Softmax turns them into probabilities that are positive and sum to 1: each probability is exp(z_i) divided by the sum of exp(z_j). The cross-entropy loss for the true class is the negative natural logarithm of the probability assigned to it, so it is small when the model is confident and right and large when it is confident and wrong.
exp overflows for large inputs: exp(1000) is infinity in double precision. Subtracting the largest logit from every logit before exponentiating gives the same probabilities, because the common factor cancels, and it keeps every exponent at or below zero.
Explore the dry run
#include <algorithm>#include <cmath>#include <iomanip>#include <iostream>#include <vector> std::vector<double> softmax(const std::vector<double>& z) { double largest = *std::max_element(z.begin(), z.end()); std::vector<double> p(z.size()); double total = 0.0; for (std::size_t i = 0; i < z.size(); ++i) { p[i] = std::exp(z[i] - largest); total += p[i]; } for (double& v : p) { v /= total; } return p;} int main() { std::cout << std::fixed << std::setprecision(3); std::vector<double> p = softmax({2.0, 1.0, 0.1}); std::cout << p[0] << " " << p[1] << " " << p[2] << "\n"; std::cout << "loss if class 0 is true: " << -std::log(p[0]) << "\n"; std::cout << "loss if class 2 is true: " << -std::log(p[2]) << "\n"; std::vector<double> big = softmax({1000.0, 1001.0}); std::cout << big[0] << " " << big[1] << "\n";}Follow the example
Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.
View the complete dry-run table
| Step / line | What happens | State after step | Output so far |
|---|---|---|---|
| Find the largest logit so every exponent can be made ≤ 0. | largest = 2.0 | — | |
| exp(0), exp(-1), and exp(-1.9) are safe to compute. | p ∝ 1, 0.368, 0.150 | — | |
| Dividing by the total 1.518 gives probabilities that sum to 1. | p ∝ 1, 0.368, 0.150 | 0.659 0.242 0.099 | |
| The loss is small for the confident class (0.417) and large for the unlikely one (2.317). | p ∝ 1, 0.368, 0.150 | 0.659 0.242 0.099 loss if class 0 is true: 0.417 loss if class 2 is true: 2.317 | |
| Without the shift, exp(1000) would overflow. With it, the answer is the same as for {0, 1}. | p ∝ 1, 0.368, 0.150 | 0.659 0.242 0.099 loss if class 0 is true: 0.417 loss if class 2 is true: 2.317 0.269 0.731 |
Cost and common mistakes
O(k) time for k classes.
- Computing log(softmax(z)) in two steps can underflow to log(0). Frameworks use a fused log-softmax for stability.
- Softmax outputs are not guaranteed to be calibrated probabilities, even though they sum to 1.
Try it yourself
What is the softmax of equal logits {1, 1, 1, 1}?
Reveal answer and reasoning
Each probability is 0.25, because all exponents are equal.