Specialty · AILesson 3 of 6Free to read

Gradient descent · AI Foundations

How does a model improve itself step by step?

Why you need it

The formula from the last lesson only exists for simple models. Neural networks have millions of numbers and no formula, yet they're trained with one simple idea: gradient descent. Check which way makes the loss go down, take a small step that way, and repeat. Understanding it explains the settings you'll meet in every training script, like the learning rate and the number of epochs.

The intuition

Picture the loss as a valley, with the weight w as your position. You can't see the bottom, but you can feel the slope under your feet. The slope of the loss is called the gradient. Positive means the loss rises to the right, so step left; negative means step right:

w = w - learning_rate × gradient

For the loss (w − 3)² the gradient is 2 × (w − 3), and the bottom is at w = 3.

The learning rate sets the step size:

too small: progress is very slow
too large: each step overshoots the bottom, lands higher on the other side, and the loss explodes

For a line w × x + b with MSE loss, the gradients (averaged over the n points) are:

dw = (2 / n) × Σ (prediction − y) × x
db = (2 / n) × Σ (prediction − y)

One pass over the data is an epoch. Compute both gradients first, then update w and b together.

Definition

Gradient

The slope of the loss: which way, and how steeply, it rises as a weight changes.

Gradient descent

Repeatedly moving the weights a small step against the gradient to lower the loss.

Learning rate

The size of each step. Too big diverges, too small crawls.

Epoch

One pass of training over all the training examples.

Examples

Walk down a valley

Try it yourself

Each step moves w toward 3 and lowers the loss.

def loss(w):
    return (w - 3) ** 2

def grad(w):
    return 2 * (w - 3)

w = 0.0
lr = 0.1
for step in range(5):
    w = w - lr * grad(w)
    print(step + 1, round(w, 3), round(loss(w), 3))

Output

1 0.6 5.76
2 1.08 3.686
3 1.464 2.359
4 1.771 1.51
5 2.017 0.966

Step by step

L4The slope of (w − 3)². At w = 0 it's −6: the loss falls to the right.
L10Step against the slope: 0 − 0.1 × (−6) = 0.6.
L11The steps get smaller as the slope flattens near the bottom.

Three learning rates

Try it yourself

Same valley, 20 steps each. The last one explodes.

for lr in [0.1, 0.5, 1.1]:
    w = 0.0
    for _ in range(20):
        w = w - lr * 2 * (w - 3)
    print(lr, round(w, 3))

Output

0.1 2.965
0.5 3.0
1.1 -112.013

Step by step

L1Try a small, a just-right and a too-large step.
L50.1 gets close; 0.5 lands exactly on 3; 1.1 overshoots further every time.

Train a line

Try it yourself

Gradient descent finds the same line as the formula: w = 2, b = 1.

xs = [1, 2, 3, 4]
ys = [3, 5, 7, 9]
w, b = 0.0, 0.0
lr = 0.05
n = len(xs)
for epoch in range(1, 1001):
    dw = sum(2 * (w * x + b - y) * x for x, y in zip(xs, ys)) / n
    db = sum(2 * (w * x + b - y) for x, y in zip(xs, ys)) / n
    w, b = w - lr * dw, b - lr * db
    if epoch in (1, 10, 100, 1000):
        print(epoch, round(w, 3), round(b, 3))

Output

1 1.75 0.6
10 2.084 0.754
100 2.022 0.937
1000 2.0 1.0

Step by step

L7How the loss changes as w changes, averaged over the points.
L9Update both at once, using gradients from the same w and b.
L10Print a few epochs to watch it settle.

Common mistakes

A learning rate that's too large

The loss gets bigger every epoch, or turns into nan. Divide the learning rate by 10 and try again.

Features on different scales

A feature in the thousands has a huge gradient and forces a tiny learning rate on everything. Scale the features first (lesson 1).

Updating one weight before computing the other's gradient

If w changes before db is computed, db uses the wrong w. Compute all gradients, then update.

Quick quiz

1. What does this print?

w = 10
for _ in range(2):
    w = w - 0.25 * 2 * (w - 2)
print(w)

A
```
4.0
```
B
```
6.0
```
C
```
2.0
```

Show the answer

Answer: A

10 − 0.5 × 8 = 6.0; 6 − 0.5 × 4 = 4.0.

2. The loss jumps around and grows bigger every epoch. What's the most likely cause?

AThe learning rate is too large
BThe learning rate is too small
CThere are too many epochs

Show the answer

Answer: A

Each step overshoots the bottom by more than the last.

Practice

3 exercises in three steps: predict what a program prints, change a program to meet a new requirement, then write one from scratch. Your code runs in the browser and is checked the moment you submit.

PredictThree steps
Type exactly what this prints.
ModifyUphill
descend should walk toward 3, the bottom of (w − 3)², but it runs away and never comes back. Fix the update.
WriteTrain it yourself
Write train(xs, ys, lr, epochs) that fits w × x + b by gradient descent and returns (w, b).
- start from w = 0 and b = 0
- each epoch, compute dw and db with the formulas from this lesson, both before changing either
- then w -= lr * dw and b -= lr * db
```
train([1], [2], 0.25, 1)   # (1.0, 1.0)
```

Practise this lesson

Open the interactive lesson to edit and run these examples and solve the exercises with instant feedback. Free to start, no sign-up.

Open the interactive lesson Course map

Interactive lessons open in order. If you have not done the earlier ones yet, you start at the course map.

Terms in this lesson

Term	Chinese	Definition
Gradient	梯度	The slope of the loss with respect to a weight.
Gradient descent	梯度下降	Lowering the loss by stepping against the gradient.
Learning rate	学习率	How big each gradient step is.
Epoch	轮次	One pass over the training data.