C++: stable softmax and cross-entropy loss

Before you start

C++ vectors and <cmath>.

Turn raw scores into probabilities, measure the loss, and avoid overflow.
Predict the sample output and explain the state changes.
Complete the practice task and check the boundary cases.

The concept

A classifier network ends with a vector of raw scores called logits. Softmax turns them into probabilities that are positive and sum to 1: each probability is exp(z_i) divided by the sum of exp(z_j). The cross-entropy loss for the true class is the negative natural logarithm of the probability assigned to it, so it is small when the model is confident and right and large when it is confident and wrong.

exp overflows for large inputs: exp(1000) is infinity in double precision. Subtracting the largest logit from every logit before exponentiating gives the same probabilities, because the common factor cancels, and it keeps every exponent at or below zero.

Code example

C++

#include <algorithm>
#include <cmath>
#include <iomanip>
#include <iostream>
#include <vector>

std::vector<double> softmax(const std::vector<double>& z) {
  double largest = *std::max_element(z.begin(), z.end());
  std::vector<double> p(z.size());
  double total = 0.0;
  for (std::size_t i = 0; i < z.size(); ++i) {
    p[i] = std::exp(z[i] - largest);
    total += p[i];
  }
  for (double& v : p) {
    v /= total;
  }
  return p;
}

int main() {
  std::cout << std::fixed << std::setprecision(3);
  std::vector<double> p = softmax({2.0, 1.0, 0.1});
  std::cout << p[0] << " " << p[1] << " " << p[2] << "\n";
  std::cout << "loss if class 0 is true: " << -std::log(p[0]) << "\n";
  std::cout << "loss if class 2 is true: " << -std::log(p[2]) << "\n";

  std::vector<double> big = softmax({1000.0, 1001.0});
  std::cout << big[0] << " " << big[1] << "\n";
}

Output

0.659 0.242 0.099
loss if class 0 is true: 0.417
loss if class 2 is true: 2.317
0.269 0.731

Example details

Follow the code

Open the walkthrough to inspect the example one step at a time.

Show guided walkthrough

Guided walkthroughReady

1#include <algorithm>

2#include <cmath>

3#include <iomanip>

4#include <iostream>

5#include <vector>

7std::vector<double> softmax(const std::vector<double>& z) {

8 double largest = *std::max_element(z.begin(), z.end());

9 std::vector<double> p(z.size());

10 double total = 0.0;

11 for (std::size_t i = 0; i < z.size(); ++i) {

12 p[i] = std::exp(z[i] - largest);

13 total += p[i];

14 }

15 for (double& v : p) {

16 v /= total;

17 }

18 return p;

19}

21int main() {

22 std::cout << std::fixed << std::setprecision(3);

23 std::vector<double> p = softmax({2.0, 1.0, 0.1});

24 std::cout << p[0] << " " << p[1] << " " << p[2] << "\n";

25 std::cout << "loss if class 0 is true: " << -std::log(p[0]) << "\n";

26 std::cout << "loss if class 2 is true: " << -std::log(p[2]) << "\n";

28 std::vector<double> big = softmax({1000.0, 1001.0});

29 std::cout << big[0] << " " << big[1] << "\n";

30}

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
Find the largest logit so every exponent can be made ≤ 0.	`largest = 2.0`	—
exp(0), exp(-1), and exp(-1.9) are safe to compute.	`p ∝ 1, 0.368, 0.150`	—
Dividing by the total 1.518 gives probabilities that sum to 1.	`p ∝ 1, 0.368, 0.150`	0.659 0.242 0.099
The loss is small for the confident class (0.417) and large for the unlikely one (2.317).	`p ∝ 1, 0.368, 0.150`	0.659 0.242 0.099 loss if class 0 is true: 0.417 loss if class 2 is true: 2.317
Without the shift, exp(1000) would overflow. With it, the answer is the same as for {0, 1}.	`p ∝ 1, 0.368, 0.150`	0.659 0.242 0.099 loss if class 0 is true: 0.417 loss if class 2 is true: 2.317 0.269 0.731

Behind the scenes

How C++ processes this code

The C++ compiler produces a native executable. Execution enters main, where objects and function calls perform the work.

Cost and common mistakes

O(k) time for k classes.

Computing log(softmax(z)) in two steps can underflow to log(0). Frameworks use a fused log-softmax for stability.
Softmax outputs are not guaranteed to be calibrated probabilities, even though they sum to 1.

Try it yourself

What is the softmax of equal logits {1, 1, 1, 1}?

Reveal answer and reasoning

Each probability is 0.25, because all exponents are equal.