JavaScript: checking a gradient numerically

Compare a hand-derived gradient with a finite-difference estimate, the standard test for backpropagation code.

const sigmoid = (z) => 1 / (1 + Math.exp(-z));

const x = 1.5;
const target = 1;
const b = -0.3;

function loss(w) {
  const a = sigmoid(w * x + b);
  return (a - target) ** 2;
}

function analyticGradient(w) {
  const a = sigmoid(w * x + b);
  return 2 * (a - target) * a * (1 - a) * x;
}

const w = 0.4;
const h = 1e-5;
const numeric = (loss(w + h) - loss(w - h)) / (2 * h);
const analytic = analyticGradient(w);

console.log("analytic: " + analytic.toFixed(8));
console.log("numeric:  " + numeric.toFixed(8));
console.log("close enough: " + (Math.abs(analytic - numeric) < 1e-8));

analytic: -0.31209319
numeric:  -0.31209319
close enough: true

Explanation

When you implement backpropagation by hand it is easy to make a mistake in a derivative, and the network may still appear to train. A gradient check compares the analytic gradient with a numerical estimate from the definition of the derivative.

The central difference (L(w + h) − L(w − h)) / 2h approximates dL/dw with error proportional to h². For a single sigmoid neuron a = σ(w·x + b) with squared error L = (a − t)², the chain rule gives dL/dw = 2(a − t)·a·(1 − a)·x. If the two numbers agree to many digits, the derivation is probably right.

—

—

—

analytic: -0.31209319
numeric:  -0.31209319
close enough: true

Cost and common mistakes

The numeric check costs two forward passes per parameter, O(P) passes for P parameters.

• If h is too small, rounding error dominates. If it is too large, the estimate is inaccurate. Values near 1e-5 are typical in double precision.
• Run the check on a few random inputs and then switch it off; it needs two extra loss evaluations per parameter.
• Non-smooth functions such as ReLU at 0 can make the check disagree at those points even when the code is right.

Try it yourself

What would a large difference between the two values mean?

Further reading

• MDN JavaScript guide (external)
• Node.js API documentation (external)
• Deep Learning (Goodfellow, Bengio, Courville) (external)