C++ · Course navigation

C++: linear regression with gradient descent

Fit y = w·x + b by repeatedly moving the parameters against the gradient of the mean squared error.

On this page

Before you start

C++ vectors and floating-point arithmetic.

Fit y = w·x + b by repeatedly moving the parameters against the gradient of the mean squared error.
Predict the sample output and explain the state changes.
Complete the practice task and check the boundary cases.

The concept

Linear regression finds the line that best fits the data by minimizing the mean squared error. Gradient descent starts from an initial guess and repeatedly nudges each parameter in the direction that reduces the error. The size of each nudge is the learning rate times the gradient.

For the model ŷ = w·x + b with error e = ŷ − y, the gradients of the mean squared error are the average of 2·e·x for w and the average of 2·e for b. The data follow y = 3x − 2, so the parameters should approach w = 3 and b = −2.

Code example

C++

#include <iomanip>
#include <iostream>
#include <vector>

int main() {
  std::vector<double> x = {0, 1, 2, 3, 4};
  std::vector<double> y = {-2, 1, 4, 7, 10};
  double w = 0.0, b = 0.0, rate = 0.05;
  double n = static_cast<double>(x.size());

  for (int epoch = 0; epoch < 5000; ++epoch) {
    double gw = 0.0, gb = 0.0;
    for (std::size_t i = 0; i < x.size(); ++i) {
      double error = w * x[i] + b - y[i];
      gw += 2 * error * x[i] / n;
      gb += 2 * error / n;
    }
    w -= rate * gw;
    b -= rate * gb;
  }
  std::cout << std::fixed << std::setprecision(2);
  std::cout << "w=" << w << " b=" << b << "\n";
  std::cout << "prediction for x=10: " << w * 10 + b << "\n";
}

Output

w=3.00 b=-2.00
prediction for x=10: 28.00

Example details

Follow the code

Open the walkthrough to inspect the example one step at a time.

Show guided walkthrough

Guided walkthroughReady

1#include <iomanip>

2#include <iostream>

3#include <vector>

5int main() {

6 std::vector<double> x = {0, 1, 2, 3, 4};

7 std::vector<double> y = {-2, 1, 4, 7, 10};

8 double w = 0.0, b = 0.0, rate = 0.05;

9 double n = static_cast<double>(x.size());

11 for (int epoch = 0; epoch < 5000; ++epoch) {

12 double gw = 0.0, gb = 0.0;

13 for (std::size_t i = 0; i < x.size(); ++i) {

14 double error = w * x[i] + b - y[i];

15 gw += 2 * error * x[i] / n;

16 gb += 2 * error / n;

17 }

18 w -= rate * gw;

19 b -= rate * gb;

20 }

21 std::cout << std::fixed << std::setprecision(2);

22 std::cout << "w=" << w << " b=" << b << "\n";

23 std::cout << "prediction for x=10: " << w * 10 + b << "\n";

24}

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
Start from w = 0, b = 0 with a small learning rate.	`w = 0, b = 0`	—
Prediction minus target for one example.	`w = 0, b = 0`	—
Move each parameter a little against its gradient.	`w → 3, b → -2`	—
After 5000 updates the fitted line matches the data.	`w ≈ 3.00, b ≈ -2.00`	w=3.00 b=-2.00
The model extrapolates: 3·10 − 2.	`w ≈ 3.00, b ≈ -2.00`	w=3.00 b=-2.00 prediction for x=10: 28.00

Behind the scenes

How C++ processes this code

The C++ compiler produces a native executable. Execution enters main, where objects and function calls perform the work.

Cost and common mistakes

O(epochs · n) time for n examples, and O(1) extra space.

A learning rate that is too high makes the parameters oscillate or diverge; too low makes progress very slow.
Because this data lies exactly on a line, the error reaches almost zero. Real data has noise, so the fit is only approximate.
Extrapolating far beyond the training range is unreliable even for a well-fitted model.

Try it yourself

What does the model predict for x = 5?

Reveal answer and reasoning

3·5 − 2 = 13, because the fitted line is y = 3x − 2.