JavaScript: a forward pass through two dense layers
Multiply by weight matrices, add biases, apply ReLU, and finish with softmax probabilities.
const relu = (z) => Math.max(0, z);
function dense(input, weights, biases, activation) {
return weights.map((row, j) => activation(row.reduce((sum, w, i) => sum + w * input[i], biases[j])));
}
function softmax(scores) {
const largest = Math.max(...scores);
const exps = scores.map((s) => Math.exp(s - largest));
const total = exps.reduce((a, b) => a + b, 0);
return exps.map((e) => e / total);
}
const input = [1, 2];
const w1 = [[0.5, -0.2], [0.3, 0.8], [-0.6, 0.1]];
const b1 = [0.1, 0.0, 0.2];
const w2 = [[1.0, -1.0, 0.5], [0.2, 0.4, -0.3], [-0.5, 0.9, 0.1]];
const b2 = [0.0, 0.1, -0.1];
const hidden = dense(input, w1, b1, relu);
const scores = dense(hidden, w2, b2, (z) => z);
const probabilities = softmax(scores);
console.log("hidden: " + hidden.map((h) => h.toFixed(2)).join(" "));
console.log("scores: " + scores.map((s) => s.toFixed(3)).join(" "));
console.log("probabilities: " + probabilities.map((p) => p.toFixed(3)).join(" "));
console.log("sum: " + probabilities.reduce((a, b) => a + b, 0).toFixed(3));hidden: 0.20 1.90 0.00 scores: -1.700 0.900 1.510 probabilities: 0.025 0.343 0.631 sum: 1.000
Explanation
A dense (fully connected) layer computes output = activation(W·x + b). Each output unit has one weight for every input, plus a bias. Stacking layers with non-linear activations lets the network represent far more than a single linear map.
The hidden layer here uses ReLU, max(0, z). The output layer produces three scores, and softmax converts them to probabilities that are positive and sum to 1. To stay numerically stable, softmax subtracts the largest score before exponentiating. The input has two features, the hidden layer has three units, and there are three output classes.
Explore the dry run
const relu = (z) => Math.max(0, z); function dense(input, weights, biases, activation) { return weights.map((row, j) => activation(row.reduce((sum, w, i) => sum + w * input[i], biases[j])));} function softmax(scores) { const largest = Math.max(...scores); const exps = scores.map((s) => Math.exp(s - largest)); const total = exps.reduce((a, b) => a + b, 0); return exps.map((e) => e / total);} const input = [1, 2];const w1 = [[0.5, -0.2], [0.3, 0.8], [-0.6, 0.1]];const b1 = [0.1, 0.0, 0.2];const w2 = [[1.0, -1.0, 0.5], [0.2, 0.4, -0.3], [-0.5, 0.9, 0.1]];const b2 = [0.0, 0.1, -0.1]; const hidden = dense(input, w1, b1, relu);const scores = dense(hidden, w2, b2, (z) => z);const probabilities = softmax(scores); console.log("hidden: " + hidden.map((h) => h.toFixed(2)).join(" "));console.log("scores: " + scores.map((s) => s.toFixed(3)).join(" "));console.log("probabilities: " + probabilities.map((p) => p.toFixed(3)).join(" "));console.log("sum: " + probabilities.reduce((a, b) => a + b, 0).toFixed(3));Follow the example
Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.
View the complete dry-run table
| Step / line | What happens | State after step | Output so far |
|---|---|---|---|
| Each output unit is the activation of a weighted sum of the inputs plus its bias. | — | — | |
| The hidden layer: the third unit's pre-activation is −0.6 + 0.2 + 0.2 = −0.2, so ReLU sets it to 0. | hidden = [0.2, 1.9, 0] | — | |
| Print the hidden activations. | hidden = [0.2, 1.9, 0] | hidden: 0.20 1.90 0.00 | |
| The output layer is linear; these raw scores are the logits. | hidden = [0.2, 1.9, 0] | hidden: 0.20 1.90 0.00 | |
| Softmax turns the logits into probabilities; the third class gets the highest. | hidden = [0.2, 1.9, 0] | hidden: 0.20 1.90 0.00 | |
| The probabilities add up to 1. | hidden = [0.2, 1.9, 0] | hidden: 0.20 1.90 0.00 scores: -1.700 0.900 1.510 probabilities: 0.025 0.343 0.631 sum: 1.000 |
Cost and common mistakes
O(weights): about the number of weights in all layers per input.
- Weights here are made up for illustration. In practice they are learned by backpropagation.
- Without the max-subtraction, softmax of large scores overflows to Infinity and gives NaN.
- Matrix code in plain arrays is slow for big networks; real libraries use optimized tensors and GPUs.
Try it yourself
Which output class has the highest probability?
Reveal answer and reasoning
The class whose score is largest. Softmax preserves the ranking of the raw scores.