C: linear regression with gradient descent

Fit y = w·x + b by repeatedly stepping the parameters against the gradient of the mean squared error.

#include <stdio.h>

int main(void) {
  double x[] = {0, 1, 2, 3, 4};
  double y[] = {-2, 1, 4, 7, 10};
  int n = 5;
  double w = 0.0, b = 0.0, rate = 0.05;

  for (int epoch = 0; epoch < 5000; epoch++) {
    double gw = 0.0, gb = 0.0;
    for (int i = 0; i < n; i++) {
      double error = w * x[i] + b - y[i];
      gw += 2.0 * error * x[i] / n;
      gb += 2.0 * error / n;
    }
    w -= rate * gw;
    b -= rate * gb;
  }
  printf("w=%.2f b=%.2f\n", w, b);
  printf("prediction for x=10: %.2f\n", w * 10 + b);
  return 0;
}

Output

w=3.00 b=-2.00
prediction for x=10: 28.00

Explanation

Linear regression fits the line y = w·x + b that minimizes the mean squared error between predictions and targets. Gradient descent starts from an initial guess and repeatedly moves each parameter a small step in the direction that lowers the error.

For the error e = w·x + b − y, the gradient of the mean squared error is the average of 2·e·x with respect to w and the average of 2·e with respect to b. The learning rate scales each step. The sample data follow y = 3x − 2, so the result should approach w = 3 and b = −2.

Explore the dry run

Guided walkthroughReady

1#include <stdio.h>

3int main(void) {

4 double x[] = {0, 1, 2, 3, 4};

5 double y[] = {-2, 1, 4, 7, 10};

6 int n = 5;

7 double w = 0.0, b = 0.0, rate = 0.05;

9 for (int epoch = 0; epoch < 5000; epoch++) {

10 double gw = 0.0, gb = 0.0;

11 for (int i = 0; i < n; i++) {

12 double error = w * x[i] + b - y[i];

13 gw += 2.0 * error * x[i] / n;

14 gb += 2.0 * error / n;

15 }

16 w -= rate * gw;

17 b -= rate * gb;

18 }

19 printf("w=%.2f b=%.2f\n", w, b);

20 printf("prediction for x=10: %.2f\n", w * 10 + b);

21 return 0;

22}

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
Start from a flat line with a small learning rate.	`w = 0, b = 0`	—
Prediction minus target for one example.	`w = 0, b = 0`	—
Move each parameter against its gradient.	`w → 3, b → -2`	—
After 5000 updates the line matches the data.	`w ≈ 3.00, b ≈ -2.00`	w=3.00 b=-2.00
3·10 − 2 for an unseen input.	`w ≈ 3.00, b ≈ -2.00`	w=3.00 b=-2.00 prediction for x=10: 28.00

Cost and common mistakes

O(epochs · n) time and O(1) extra space.

Dividing an int by an int truncates. The code divides by n in a double expression, so the average is exact.
A learning rate that is too high makes w and b grow without limit instead of settling.
Because the data lie exactly on a line, the error goes to nearly zero. Real data is noisy.

Try it yourself

What value does the model predict for x = 5?

Reveal answer and reasoning

3·5 − 2 = 13, because the fitted line is y = 3x − 2.

C: linear regression with gradient descent

Explanation

Follow the example

Cost and common mistakes

Try it yourself

Further reading