Lesson · ⏱ 6 min read

🏭 Training vs inference: how ML runs in production

Q: Scoring all 2 million customers for churn risk every night is…

Batch inference. It's prediction (inference) done on a schedule over many items, which is batch.

🏠 Real-life example

A chef spends months perfecting a recipe in a test kitchen. That's training: slow, expensive, done occasionally. Then the restaurant serves that dish to hundreds of customers a day, quickly and consistently. That's inference. Running AI in production is mostly about the restaurant, not the test kitchen.

💡 In simple words

Training uses lots of data and compute, often GPUs, to produce a model artifact: a saved file of learned parameters. Inference loads that artifact and makes predictions on new inputs, either in real time (an API answering in milliseconds) or in batches (scoring a million customers overnight). Inference has different priorities from training: latency, throughput, cost per request and reliability. Models are versioned like code, so you know exactly which model gave which answer. Infrastructure engineers usually own the inference side.

🪜 Step by step

Training happens occasionally (daily, weekly) on lots of past data, often on powerful machines.
The result is a saved model file plus its preprocessing steps, stored with a version.
Inference uses that model to predict for new inputs, either in batches (score all customers nightly) or in real time (per request).
Real time inference must be fast and reliable, so it runs behind an API with monitoring.
Over time data changes, so you monitor results and retrain with fresh data.

🔑 Key terms

Model artifact: The saved, trained model file that gets deployed.
Inference: Using a trained model to make predictions.
Batch vs real-time: Scoring many items on a schedule vs answering each request instantly.

👀 See it in action

text

Training (weekly):   data → train.py on GPU → fraud_model_v12.pkl → registry
Inference (24×7):    UPI payment → API loads v12 → P(fraud)=0.03 → approve
                     latency budget: < 50 ms per payment

Same model, two very different engineering problems.

⚠️ Common mistakes to avoid

Saving the model without its preprocessing. Predictions break or silently go wrong in production.
Building real time serving when a nightly batch would do. Batch is simpler and cheaper.

✅ Quick check

Scoring all 2 million customers for churn risk every night is…

Batch inference
Real-time inference
Training

Show answer

A. Batch inference: It's prediction (inference) done on a schedule over many items, which is batch.

🙋 Questions learners ask

Which costs more, training or inference?

For most companies, inference, because it runs constantly for every user. For LLMs, inference cost is the main bill unless you train your own models.

Does a model keep learning while in use?

Usually no. It's fixed until you retrain it. Online learning exists but is rarer and riskier.

📌 Remember

Training creates the model. Inference serves it, fast, cheaply and reliably.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

ML Engineer · MLOps / AI Platform Engineer

Continue in the course