C++: linear regression with gradient descent

Fit y = w·x + b by repeatedly moving the parameters against the gradient of the mean squared error.

C++

#include <iomanip>
#include <iostream>
#include <vector>

int main() {
  std::vector<double> x = {0, 1, 2, 3, 4};
  std::vector<double> y = {-2, 1, 4, 7, 10};
  double w = 0.0, b = 0.0, rate = 0.05;
  double n = static_cast<double>(x.size());

  for (int epoch = 0; epoch < 5000; ++epoch) {
    double gw = 0.0, gb = 0.0;
    for (std::size_t i = 0; i < x.size(); ++i) {
      double error = w * x[i] + b - y[i];
      gw += 2 * error * x[i] / n;
      gb += 2 * error / n;
    }
    w -= rate * gw;
    b -= rate * gb;
  }
  std::cout << std::fixed << std::setprecision(2);
  std::cout << "w=" << w << " b=" << b << "\n";
  std::cout << "prediction for x=10: " << w * 10 + b << "\n";
}

Output

w=3.00 b=-2.00
prediction for x=10: 28.00

Explanation

Linear regression finds the line that best fits the data by minimizing the mean squared error. Gradient descent starts from an initial guess and repeatedly nudges each parameter in the direction that reduces the error. The size of each nudge is the learning rate times the gradient.

For the model ŷ = w·x + b with error e = ŷ − y, the gradients of the mean squared error are the average of 2·e·x for w and the average of 2·e for b. The data follow y = 3x − 2, so the parameters should approach w = 3 and b = −2.

Explore the dry run

Guided walkthroughReady

1#include <iomanip>

2#include <iostream>

3#include <vector>

5int main() {

6 std::vector<double> x = {0, 1, 2, 3, 4};

7 std::vector<double> y = {-2, 1, 4, 7, 10};

8 double w = 0.0, b = 0.0, rate = 0.05;

9 double n = static_cast<double>(x.size());

11 for (int epoch = 0; epoch < 5000; ++epoch) {

12 double gw = 0.0, gb = 0.0;

13 for (std::size_t i = 0; i < x.size(); ++i) {

14 double error = w * x[i] + b - y[i];

15 gw += 2 * error * x[i] / n;

16 gb += 2 * error / n;

17 }

18 w -= rate * gw;

19 b -= rate * gb;

20 }

21 std::cout << std::fixed << std::setprecision(2);

22 std::cout << "w=" << w << " b=" << b << "\n";

23 std::cout << "prediction for x=10: " << w * 10 + b << "\n";

24}

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
Start from w = 0, b = 0 with a small learning rate.	`w = 0, b = 0`	—
Prediction minus target for one example.	`w = 0, b = 0`	—
Move each parameter a little against its gradient.	`w → 3, b → -2`	—
After 5000 updates the fitted line matches the data.	`w ≈ 3.00, b ≈ -2.00`	w=3.00 b=-2.00
The model extrapolates: 3·10 − 2.	`w ≈ 3.00, b ≈ -2.00`	w=3.00 b=-2.00 prediction for x=10: 28.00

Cost and common mistakes

O(epochs · n) time for n examples, and O(1) extra space.

A learning rate that is too high makes the parameters oscillate or diverge; too low makes progress very slow.
Because this data lies exactly on a line, the error reaches almost zero. Real data has noise, so the fit is only approximate.
Extrapolating far beyond the training range is unreliable even for a well-fitted model.

Try it yourself

What does the model predict for x = 5?

Reveal answer and reasoning

3·5 − 2 = 13, because the fitted line is y = 3x − 2.

C++: linear regression with gradient descent

Explanation

Follow the example

Cost and common mistakes

Try it yourself

Further reading