Activation functions and softmax
An activation function is the nonlinearity applied to a neuron's weighted sum. Without it, a stack of layers would collapse into one linear transformation, no matter how many there are. The sigmoid σ(z) = 1 / (1 + e⁻ᶻ) squashes values into (0, 1). The hyperbolic tangent squashes into (−1, 1) and is centred on zero. ReLU, max(0, z), is zero for negative inputs and the identity for positive ones.
Python
import math
def sigmoid(z):
return 1 / (1 + math.exp(-z))
def relu(z):
return max(0.0, z)
def softmax(values):
peak = max(values)
exps = [math.exp(v - peak) for v in values]
total = sum(exps)
return [e / total for e in exps]
for z in (-2, 0, 2):
print(z, round(sigmoid(z), 3), round(math.tanh(z), 3), relu(z))
probabilities = softmax([2.0, 1.0, 0.1])
print([round(p, 3) for p in probabilities], round(sum(probabilities), 6))
print([round(p, 3) for p in softmax([1000, 1001])])