Lesson · ⏱ 7 min read

📡 Monitoring models: drift and silent failures

Q: Model accuracy slowly drops over months with no code changes. Most likely cause?

Drift: the real-world data or relationships changed. When the world changes, a frozen model gets stale. That's drift.

🏠 Real-life example

A demand-forecasting model trained before Diwali suddenly sees completely different shopping patterns during the festival season. Nothing 'crashed', but its predictions are now wrong. Like a car slowly going out of alignment, models drift as the world changes, and you only notice if you keep checking.

💡 In simple words

Software errors are loud. Model errors are often silent. Data drift means the inputs change: new user types, new products, a changed upstream field. Concept drift means the relationship changes: what used to signal fraud no longer does. Monitor input distributions, prediction distributions, data quality and, when labels arrive, real accuracy. For LLM apps, trace every request (prompt, retrieved docs, output, cost, latency) and track user feedback. Set alerts and a retraining or rollback plan before you need them.

🎬 See how it works

Nothing crashed, but the inputs drifted during the festival season. Monitoring catches the line crossing the alert threshold before accuracy quietly falls apart.

🪜 Step by step

System health: latency, errors and throughput, with alerts.
Data: compare live feature distributions with training data (for example with PSI); watch for missing values and schema changes.
Predictions: watch the share of each predicted class and the confidence scores over time.
Real performance: when true labels arrive (days or weeks later), compute accuracy, precision and recall by week.
Act: each alert has an owner and an action, such as investigate the pipeline, retrain or roll back.

🔑 Key terms

Data drift: Input data looks different from the training data.
Concept drift: The real-world relationship the model learned has changed.
Tracing: Recording each request's steps, inputs and outputs for debugging.

👀 See it in action

text

Weekly monitor — fraud model v12
  avg transaction amount   ₹1,240 → ₹2,980   ⚠️ data drift (festival season)
  % flagged as fraud       1.1%  → 4.6%     ⚠️ prediction shift
  confirmed-fraud recall   0.81  → 0.62     🚨 alert → retrain with recent data

Watch the inputs, the outputs and, when you can, the true outcomes.

⚠️ Common mistakes to avoid

Monitoring only uptime. The service can be up while predictions are silently wrong.
Too many noisy alerts, so people ignore them. Alert only on what needs action.

✅ Quick check

Model accuracy slowly drops over months with no code changes. Most likely cause?

Too many tests
A syntax error
Drift: the real-world data or relationships changed

Show answer

C. Drift: the real-world data or relationships changed: When the world changes, a frozen model gets stale. That's drift.

🙋 Questions learners ask

How often should I retrain?

When monitoring shows drift or falling performance, or on a regular schedule if your data changes predictably. Always evaluate before replacing a model.

What tools are used?

Prometheus and Grafana for system metrics, Evidently or similar for drift, and your warehouse for performance reports.

📌 Remember

Models decay silently. Monitor inputs, outputs and outcomes, with alerts.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

ML Engineer · MLOps / AI Platform Engineer

Continue in the course