AI Path Finder
Free consultation

Lesson · ⏱ 7 min read

📝 Train/test split and overfitting

EnglishHinglish

🏠 Real-life example

A student who memorises last year's question paper word for word scores 100% on that paper, then fails the real exam with new questions. That's overfitting. Teachers prevent it by keeping the real exam questions secret. In ML we do the same: we hide a 'test set' the model never sees during training.

💡 In simple words

Split your data before training: typically 70–80% for training and 20–30% for testing. The model learns only from the training set, and you judge it only on the test set, which shows how it will do on new, real-world data. Overfitting means great training scores but poor test scores: the model memorised noise instead of learning patterns. Underfitting means poor scores on both: the model is too simple. Cross-validation repeats the split several times for a more reliable estimate.

🎬 See how it works

Train · 80% · the model studies theseTest · 20% 🔒Train score 0.99Test score 0.71Big gap → overfitting (memorised, not learned)

Split the data first: the model learns from 80% and is judged on the 20% it never saw. A big gap between the two scores means overfitting.

🪜 Step by step

  1. Split before anything else: about 70% train, 15% validation, 15% test.
  2. Train on the training set only, and fit cleaning steps (like scaling) on it too.
  3. Compare models and settings on the validation set.
  4. Watch the gap: if training is much better than validation, you're overfitting. Simplify or add data.
  5. Use the test set once, at the end, for the number you report.

🔑 Key terms

Test set
Data hidden during training and used only for the final check.
Overfitting
Memorising the training data and failing on new data.
Cross-validation
Repeating train/test splits to get a more trustworthy score.

👀 See it in action

python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model.fit(X_train, y_train)
print("train:", model.score(X_train, y_train))   # 0.99
print("test: ", model.score(X_test, y_test))     # 0.71  ← overfitting alarm 🚨

A big gap between train and test scores is the classic overfitting signal.

⚠️ Common mistakes to avoid

  • Random splits for time based data. Train on the past, test on the future, like in real use.
  • Same customer in train and test. The model memorises the customer; split by customer instead.
  • Checking the test score again and again while tuning. It stops being an honest test.

✅ Quick check

Training accuracy is 98% and test accuracy is 65%. What's happening?

  1. The model is perfect
  2. Underfitting
  3. Overfitting
Show answer

C. Overfitting: It does brilliantly on data it has seen and poorly on new data, which means it memorised rather than learned.

🙋 Questions learners ask

What is cross validation?

Splitting training data into k parts and rotating which part validates. It gives a more stable estimate when data is small.

How do I fix underfitting?

The model is too simple: add better features, use a more powerful model, or train longer.

📌 Remember

Never judge a model on data it trained on.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

AI Foundations Path · ML Engineer · MLOps / AI Platform Engineer · AI Research Engineer

Continue in the course