AI Path Finder
Free consultation

Lesson · ⏱ 6 min read

🌲 Decision trees, random forests and boosting

EnglishHinglish

🏠 Real-life example

A bank officer decides on a loan by asking a series of questions: Is salary above ₹50k? Any past defaults? Job for more than 2 years? That flowchart is a decision tree. Asking 100 officers who each saw slightly different files, then taking a vote, is a random forest. Boosting is like a coaching class where each new tutor focuses on the questions previous students got wrong.

💡 In simple words

A decision tree splits data with simple yes/no questions, which makes it easy to explain but prone to overfitting. A random forest trains many trees on random samples and averages their votes, which is more accurate and stable. Gradient boosting (XGBoost, LightGBM) builds trees one after another, each fixing the errors of the last. On tabular business data, like spreadsheets of customers or transactions, boosted trees are often the best-performing models, beating deep learning.

🪜 Step by step

  1. A decision tree asks yes/no questions: is income > ₹50k? Has the customer missed a payment? Each answer leads to the next question, ending in a prediction.
  2. One deep tree memorises data (overfits). Limit depth or the minimum examples per leaf.
  3. Random forest: train hundreds of trees on random samples and average their votes. More stable.
  4. Gradient boosting (XGBoost, LightGBM): build small trees one after another, each fixing earlier mistakes. Usually the most accurate on tables.
  5. Tune a few key settings (number of trees, depth, learning rate) using validation data and early stopping.

🔑 Key terms

Decision tree
A flowchart of yes/no questions that leads to a prediction.
Random forest
Many trees voting together for a steadier answer.
Gradient boosting
Trees built in sequence, each correcting earlier mistakes.

👀 See it in action

python
from xgboost import XGBClassifier
model = XGBClassifier(n_estimators=300, max_depth=4, learning_rate=0.05)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))
# feature_importances_ shows which inputs mattered most

A strong default for spreadsheet-style data.

⚠️ Common mistakes to avoid

  • Too many boosting rounds without early stopping, which overfits.
  • Trusting default feature importance blindly; it can favour features with many unique values. Cross check with SHAP or permutation importance.

✅ Quick check

For predicting customer churn from a table of 30 columns, a strong first choice is…

  1. A huge deep neural network
  2. A chatbot
  3. Gradient-boosted trees
Show answer

C. Gradient-boosted trees: Boosted trees are fast, accurate and explainable on tabular data. Try them before deep learning.

🙋 Questions learners ask

Do trees need scaled features?

No. Trees split on thresholds, so scaling doesn't change them. That's one reason they're easy to use.

Why not use deep learning for tables?

On typical business tables, gradient boosting is usually as good or better, faster to train and easier to explain.

📌 Remember

Tables → try boosted trees first.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

ML Engineer

Continue in the course