C: softmax and why the largest score is subtracted
Convert scores to probabilities and show that the naive formula overflows for large scores.
On this page
C math library and arrays.
- Convert scores to probabilities and show that the naive formula overflows for large scores.
- Predict the sample output and explain the state changes.
- Complete the practice task and check the boundary cases.
The concept
Softmax turns a vector of scores into probabilities: each output is exp(z_i) divided by the sum of exp(z_j). The outputs are positive and sum to 1. A network usually ends in softmax when it must choose one of several classes.
exp grows very fast. In double precision exp(1000) overflows to infinity, and infinity divided by infinity is NaN. Because softmax(z) equals softmax(z − c) for any constant c, subtracting the largest score first keeps every exponent at or below zero and gives the same probabilities. The naive and stable versions agree for small scores and only the stable one survives large ones.
Code example
#include <math.h>
#include <stdio.h>
static void softmax_naive(const double *z, double *out, int n) {
double total = 0.0;
for (int i = 0; i < n; i++) {
out[i] = exp(z[i]);
total += out[i];
}
for (int i = 0; i < n; i++) {
out[i] /= total;
}
}
static void softmax_stable(const double *z, double *out, int n) {
double largest = z[0];
for (int i = 1; i < n; i++) {
if (z[i] > largest) {
largest = z[i];
}
}
double total = 0.0;
for (int i = 0; i < n; i++) {
out[i] = exp(z[i] - largest);
total += out[i];
}
for (int i = 0; i < n; i++) {
out[i] /= total;
}
}
int main(void) {
double small[3] = {2.0, 1.0, 0.1};
double out[3];
softmax_stable(small, out, 3);
printf("stable: %.3f %.3f %.3f\n", out[0], out[1], out[2]);
softmax_naive(small, out, 3);
printf("naive: %.3f %.3f %.3f\n", out[0], out[1], out[2]);
double big[2] = {1000.0, 1001.0};
double big_out[2];
softmax_naive(big, big_out, 2);
printf("naive on large scores is NaN: %s\n", isnan(big_out[0]) ? "yes" : "no");
softmax_stable(big, big_out, 2);
printf("stable on large scores: %.3f %.3f\n", big_out[0], big_out[1]);
return 0;
}stable: 0.659 0.242 0.099 naive: 0.659 0.242 0.099 naive on large scores is NaN: yes stable on large scores: 0.269 0.731
Follow the code
Open the walkthrough to inspect the example one step at a time.
Show guided walkthrough
#include <math.h>#include <stdio.h> static void softmax_naive(const double *z, double *out, int n) { double total = 0.0; for (int i = 0; i < n; i++) { out[i] = exp(z[i]); total += out[i]; } for (int i = 0; i < n; i++) { out[i] /= total; }} static void softmax_stable(const double *z, double *out, int n) { double largest = z[0]; for (int i = 1; i < n; i++) { if (z[i] > largest) { largest = z[i]; } } double total = 0.0; for (int i = 0; i < n; i++) { out[i] = exp(z[i] - largest); total += out[i]; } for (int i = 0; i < n; i++) { out[i] /= total; }} int main(void) { double small[3] = {2.0, 1.0, 0.1}; double out[3]; softmax_stable(small, out, 3); printf("stable: %.3f %.3f %.3f\n", out[0], out[1], out[2]); softmax_naive(small, out, 3); printf("naive: %.3f %.3f %.3f\n", out[0], out[1], out[2]); double big[2] = {1000.0, 1001.0}; double big_out[2]; softmax_naive(big, big_out, 2); printf("naive on large scores is NaN: %s\n", isnan(big_out[0]) ? "yes" : "no"); softmax_stable(big, big_out, 2); printf("stable on large scores: %.3f %.3f\n", big_out[0], big_out[1]); return 0;}Follow the example
Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.
View the complete dry-run table
| Step / line | What happens | State after step | Output so far |
|---|---|---|---|
| Every exponent is at most 0, so exp never overflows. | — | — | |
| The probabilities for scores 2.0, 1.0, 0.1. | — | stable: 0.659 0.242 0.099 | |
| For small scores the naive version gives the same answer. | — | stable: 0.659 0.242 0.099 naive: 0.659 0.242 0.099 | |
| exp(1000) is infinity, and infinity / infinity is NaN. | — | stable: 0.659 0.242 0.099 naive: 0.659 0.242 0.099 naive on large scores is NaN: yes | |
| After subtracting 1001 the scores become −1 and 0, giving the same result as for {0, 1}. | — | stable: 0.659 0.242 0.099 naive: 0.659 0.242 0.099 naive on large scores is NaN: yes stable on large scores: 0.269 0.731 |
Behind the scenes
How C processes this code
The C compiler produces a native program. Variables occupy memory, and execution enters main.
Cost and common mistakes
O(n) time for n scores.
- Link with -lm on Linux, or the call to exp fails to link.
- Printing a NaN gives platform-specific text such as nan or -nan. Test with isnan instead of printing it.
- Training code usually combines softmax and the logarithm into one stable log-softmax.
Try it yourself
What does softmax give for equal scores?
Reveal answer and reasoning
Equal probabilities: with four equal scores each probability is 0.25.