Activation functions and softmax
Compare sigmoid, tanh, and ReLU, and compute a numerically stable softmax.
On this page
Functions and the exponential.
- Compare sigmoid, tanh, and ReLU, and compute a numerically stable softmax.
- Predict the sample output and explain the state changes.
- Complete the practice task and check the boundary cases.
The concept
An activation function is the nonlinearity applied to a neuron's weighted sum. Without it, a stack of layers would collapse into one linear transformation, no matter how many there are. The sigmoid σ(z) = 1 / (1 + e⁻ᶻ) squashes values into (0, 1). The hyperbolic tangent squashes into (−1, 1) and is centred on zero. ReLU, max(0, z), is zero for negative inputs and the identity for positive ones.
Sigmoid and tanh saturate: for large positive or negative inputs their slope is almost zero, so gradients become tiny and learning in deep networks stalls (the vanishing-gradient problem). ReLU has a constant slope for positive inputs and is cheap to compute, which is why it became the default in many deep networks, though a unit stuck at negative inputs can stop learning, called a dying ReLU. Newer networks often use smoother variants such as GELU or SiLU.
Softmax turns a vector of scores (logits) into a probability distribution: each output is e^(zᵢ) divided by the sum of all e^(zⱼ). Computing it directly can overflow for large scores. Subtracting the maximum score first gives the same result, since softmax is unchanged when a constant is subtracted from every input, and the exponentials stay at most 1. The last line shows scores of 1000 and 1001 handled without overflow.
Code example
import math
def sigmoid(z):
return 1 / (1 + math.exp(-z))
def relu(z):
return max(0.0, z)
def softmax(values):
peak = max(values)
exps = [math.exp(v - peak) for v in values]
total = sum(exps)
return [e / total for e in exps]
for z in (-2, 0, 2):
print(z, round(sigmoid(z), 3), round(math.tanh(z), 3), relu(z))
probabilities = softmax([2.0, 1.0, 0.1])
print([round(p, 3) for p in probabilities], round(sum(probabilities), 6))
print([round(p, 3) for p in softmax([1000, 1001])])-2 0.119 -0.964 0.0 0 0.5 0.0 0.0 2 0.881 0.964 2 [0.659, 0.242, 0.099] 1.0 [0.269, 0.731]
Follow the code
Open the walkthrough to inspect the example one step at a time.
Show guided walkthrough
import math def sigmoid(z): return 1 / (1 + math.exp(-z)) def relu(z): return max(0.0, z) def softmax(values): peak = max(values) exps = [math.exp(v - peak) for v in values] total = sum(exps) return [e / total for e in exps] for z in (-2, 0, 2): print(z, round(sigmoid(z), 3), round(math.tanh(z), 3), relu(z))probabilities = softmax([2.0, 1.0, 0.1])print([round(p, 3) for p in probabilities], round(sum(probabilities), 6))print([round(p, 3) for p in softmax([1000, 1001])])Follow the example
Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.
View the complete dry-run table
| Step / line | What happens | State after step | Output so far |
|---|---|---|---|
| Sigmoid maps any real number to (0, 1). At z = 0 it is exactly 0.5. | — | — | |
| ReLU passes positive values unchanged and sets negative values to zero. | — | — | |
| Subtracting the largest score keeps every exponent at or below 0, so nothing overflows, and the final ratios are unchanged. | — | — | |
| Compare the three activations at z = −2, 0, and 2. Sigmoid stays positive, tanh is centred on zero, and ReLU clips the negative input. | — | -2 0.119 -0.964 0.0 0 0.5 0.0 0.0 2 0.881 0.964 2 | |
| Softmax gives positive values that sum to 1: a probability distribution over the three classes. | — | -2 0.119 -0.964 0.0 0 0.5 0.0 0.0 2 0.881 0.964 2 [0.659, 0.242, 0.099] 1.0 | |
| Scores of 1000 and 1001 give 0.269 and 0.731, the same as scores of 0 and 1. A naive version would overflow. | — | -2 0.119 -0.964 0.0 0 0.5 0.0 0.0 2 0.881 0.964 2 [0.659, 0.242, 0.099] 1.0 [0.269, 0.731] |
Behind the scenes
How Python processes this code
Python runs this source through its interpreter. It creates objects for values and resolves names as each statement runs.
Cost and common mistakes
All of these are O(1) per value. Softmax over k values is O(k).
- Without nonlinear activations, any number of layers is equivalent to one linear layer.
- Naive softmax overflows for large inputs. Always subtract the maximum, or use a library implementation.
- Sigmoid and tanh saturate for large inputs and give vanishing gradients. For hidden layers, ReLU-style activations are usually preferred.
Try it yourself
What does softmax([3, 3]) return?
Reveal answer and reasoning
[0.5, 0.5]. Equal scores give equal probabilities.