Cross-entropy loss and the softmax gradient
A classifier with softmax outputs produces a probability for each class. The loss that fits this setting is cross-entropy: for the correct class c, the loss is −log p_c. It is small when the model assigns a high probability to the right class and grows without bound as that probability approaches zero. Minimizing it is equivalent to maximum-likelihood training of the model.
Python
import math
def softmax(values):
peak = max(values)
exps = [math.exp(v - peak) for v in values]
total = sum(exps)
return [e / total for e in exps]
def cross_entropy(scores, target):
return -math.log(softmax(scores)[target])
def sigmoid(z):
return 1 / (1 + math.exp(-z))
scores = [2.0, 1.0, 0.1]
target = 0
p = softmax(scores)
analytic = [p[i] - (1 if i == target else 0) for i in range(3)]
numeric = []
for i in range(3):
up = scores[:]
up[i] += 1e-6
down = scores[:]
down[i] -= 1e-6
numeric.append((cross_entropy(up, target) - cross_entropy(down, target)) / 2e-6)
print(all(abs(a - n) < 1e-6 for a, n in zip(analytic, numeric)))
print(round(cross_entropy(scores, 0), 4), round(cross_entropy(scores, 2), 4))
y = sigmoid(-6)
print(round((y - 1) * y * (1 - y), 4), round(y - 1, 4))