Lesson · ⏱ 8 min read
⛰️ Gradients and gradient descent: how models improve
EnglishHinglish
🏠 Real-life example
Imagine you're on a foggy hill at night and need to reach the bottom. You can't see far, so you feel the slope under your feet and take a small step downhill. Then you repeat. The slope is the gradient, each step is an update, and the step size is the learning rate. That's exactly how neural networks learn.
💡 In simple words
A model's 'loss' is a number measuring how wrong its predictions are. The gradient tells you which direction each parameter should move to reduce the loss, and how strongly. Gradient descent repeatedly nudges all parameters a little in that downhill direction. If the learning rate is too big you overshoot and bounce around; too small and training takes forever. Backpropagation is the efficient method, using the chain rule from calculus, that computes gradients for millions of parameters at once. Optimisers like Adam are smarter versions of gradient descent.
🎬 See how it works
Training is a ball rolling down the loss curve: each step goes downhill, and the learning rate sets the step size. It stops at the lowest loss.
🪜 Step by step
- Start with random weights; the model's predictions are bad.
- Measure how bad with a loss function (for example, average squared error).
- Compute the gradient: for each weight, which direction makes the loss go up, and how steeply.
- Update each weight a small step in the opposite direction: weight = weight − learning_rate × gradient.
- Repeat over many mini batches until the loss stops improving on validation data.
🔑 Key terms
- Loss
- A number measuring how wrong the model is. Lower is better.
- Learning rate
- How big a step to take on each update.
- Backpropagation
- The method that computes gradients through all the layers of a network.
👀 See it in action
# Find w so that w * 3 ≈ 12 (answer: 4)
w, lr = 0.0, 0.05
for step in range(20):
pred = w * 3
loss = (pred - 12) ** 2
grad = 2 * (pred - 12) * 3 # slope of the loss
w -= lr * grad # step downhill
print(round(w, 3)) # → 4.0The whole idea of training in 7 lines: predict, measure the error, step downhill, repeat.
⚠️ Common mistakes to avoid
- Learning rate too high: loss jumps around or explodes. Too low: training crawls. Try a few values.
- Not plotting the loss curve. It's the quickest way to see if training is working.
✅ Quick check
Training loss jumps up and down wildly and never settles. A likely fix is…
- Increase the learning rate
- Delete the test set
- Decrease the learning rate
Show answer
C. Decrease the learning rate: Steps that are too big overshoot the bottom. Smaller steps settle down.
🙋 Questions learners ask
Do I need to calculate gradients by hand?
No. Frameworks like PyTorch compute them automatically (autograd). Understanding the idea is what matters.
What is Adam?
An optimiser that adapts the step size for each weight using recent gradients. It's a strong default for neural networks.
📌 Remember
Gradient = which way is downhill. Learning = many small steps down the loss hill.
📖 Words from this lesson (AI dictionary)
Learn this as part of a full course
Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:
Continue in the course