JavaScript: a forward pass through two dense layers

Before you start

JavaScript arrays and Math.exp.

Multiply by weight matrices, add biases, apply ReLU, and finish with softmax probabilities.
Predict the sample output and explain the state changes.
Complete the practice task and check the boundary cases.

The concept

A dense (fully connected) layer computes output = activation(W·x + b). Each output unit has one weight for every input, plus a bias. Stacking layers with non-linear activations lets the network represent far more than a single linear map.

The hidden layer here uses ReLU, max(0, z). The output layer produces three scores, and softmax converts them to probabilities that are positive and sum to 1. To stay numerically stable, softmax subtracts the largest score before exponentiating. The input has two features, the hidden layer has three units, and there are three output classes.

Code example

JavaScript

const relu = (z) => Math.max(0, z);

function dense(input, weights, biases, activation) {
  return weights.map((row, j) => activation(row.reduce((sum, w, i) => sum + w * input[i], biases[j])));
}

function softmax(scores) {
  const largest = Math.max(...scores);
  const exps = scores.map((s) => Math.exp(s - largest));
  const total = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / total);
}

const input = [1, 2];
const w1 = [[0.5, -0.2], [0.3, 0.8], [-0.6, 0.1]];
const b1 = [0.1, 0.0, 0.2];
const w2 = [[1.0, -1.0, 0.5], [0.2, 0.4, -0.3], [-0.5, 0.9, 0.1]];
const b2 = [0.0, 0.1, -0.1];

const hidden = dense(input, w1, b1, relu);
const scores = dense(hidden, w2, b2, (z) => z);
const probabilities = softmax(scores);

console.log("hidden: " + hidden.map((h) => h.toFixed(2)).join(" "));
console.log("scores: " + scores.map((s) => s.toFixed(3)).join(" "));
console.log("probabilities: " + probabilities.map((p) => p.toFixed(3)).join(" "));
console.log("sum: " + probabilities.reduce((a, b) => a + b, 0).toFixed(3));

Output

hidden: 0.20 1.90 0.00
scores: -1.700 0.900 1.510
probabilities: 0.025 0.343 0.631
sum: 1.000

Example details

Follow the code

Open the walkthrough to inspect the example one step at a time.

Show guided walkthrough

Guided walkthroughReady

1const relu = (z) => Math.max(0, z);

3function dense(input, weights, biases, activation) {

4 return weights.map((row, j) => activation(row.reduce((sum, w, i) => sum + w * input[i], biases[j])));

5}

7function softmax(scores) {

8 const largest = Math.max(...scores);

9 const exps = scores.map((s) => Math.exp(s - largest));

10 const total = exps.reduce((a, b) => a + b, 0);

11 return exps.map((e) => e / total);

12}

14const input = [1, 2];

15const w1 = [[0.5, -0.2], [0.3, 0.8], [-0.6, 0.1]];

16const b1 = [0.1, 0.0, 0.2];

17const w2 = [[1.0, -1.0, 0.5], [0.2, 0.4, -0.3], [-0.5, 0.9, 0.1]];

18const b2 = [0.0, 0.1, -0.1];

20const hidden = dense(input, w1, b1, relu);

21const scores = dense(hidden, w2, b2, (z) => z);

22const probabilities = softmax(scores);

24console.log("hidden: " + hidden.map((h) => h.toFixed(2)).join(" "));

25console.log("scores: " + scores.map((s) => s.toFixed(3)).join(" "));

26console.log("probabilities: " + probabilities.map((p) => p.toFixed(3)).join(" "));

27console.log("sum: " + probabilities.reduce((a, b) => a + b, 0).toFixed(3));

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
Each output unit is the activation of a weighted sum of the inputs plus its bias.	`—`	—
The hidden layer: the third unit's pre-activation is −0.6 + 0.2 + 0.2 = −0.2, so ReLU sets it to 0.	`hidden = [0.2, 1.9, 0]`	—
Print the hidden activations.	`hidden = [0.2, 1.9, 0]`	hidden: 0.20 1.90 0.00
The output layer is linear; these raw scores are the logits.	`hidden = [0.2, 1.9, 0]`	hidden: 0.20 1.90 0.00
Softmax turns the logits into probabilities; the third class gets the highest.	`hidden = [0.2, 1.9, 0]`	hidden: 0.20 1.90 0.00
The probabilities add up to 1.	`hidden = [0.2, 1.9, 0]`	hidden: 0.20 1.90 0.00 scores: -1.700 0.900 1.510 probabilities: 0.025 0.343 0.631 sum: 1.000

Behind the scenes

How JavaScript processes this code

A JavaScript engine parses and runs the source. A function body runs when the function is called. console.log writes to the console.

Cost and common mistakes

O(weights): about the number of weights in all layers per input.

Weights here are made up for illustration. In practice they are learned by backpropagation.
Without the max-subtraction, softmax of large scores overflows to Infinity and gives NaN.
Matrix code in plain arrays is slow for big networks; real libraries use optimized tensors and GPUs.

Try it yourself

Which output class has the highest probability?

Reveal answer and reasoning

The class whose score is largest. Softmax preserves the ranking of the raw scores.